Jul 30 00:15:41.069450 ip-10-0-135-68 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 30 00:15:41.069463 ip-10-0-135-68 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 30 00:15:41.069470 ip-10-0-135-68 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 30 00:15:41.069705 ip-10-0-135-68 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 30 00:15:51.206534 ip-10-0-135-68 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 30 00:15:51.206553 ip-10-0-135-68 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 74560bae19ba47489fff0861d01fe01d --
Jul 30 00:17:54.562515 ip-10-0-135-68 systemd[1]: Starting Kubernetes Kubelet...
Jul 30 00:17:55.014583 ip-10-0-135-68 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 00:17:55.014583 ip-10-0-135-68 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 30 00:17:55.014583 ip-10-0-135-68 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 00:17:55.014583 ip-10-0-135-68 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 30 00:17:55.014583 ip-10-0-135-68 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 00:17:55.015716 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.015522    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 30 00:17:55.020663 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020645    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 00:17:55.020663 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020662    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020666    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020670    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020673    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020677    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020682    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020685    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020688    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020691    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020694    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020697    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020700    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020703    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020706    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020709    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020712    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020715    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020718    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020721    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020724    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 00:17:55.020731 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020727    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020730    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020733    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020736    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020739    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020742    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020745    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020748    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020751    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020754    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020757    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020760    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020763    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020765    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020768    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020771    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020774    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020777    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020780    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020783    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 00:17:55.021271 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020785    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020788    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020791    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020794    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020796    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020799    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020802    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020804    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020807    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020810    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020821    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020824    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020827    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020830    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020833    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020836    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020839    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020842    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020845    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020847    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 00:17:55.021802 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020850    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 00:17:55.022332 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020855    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 00:17:55.022332 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020860    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 00:17:55.022332 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020864    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 00:17:55.022332 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020867    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 00:17:55.022332 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020871    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 00:17:55.022332 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020874    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 00:17:55.022332 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020877    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 00:17:55.022332 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020880    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 00:17:55.022332 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020884    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 00:17:55.022332 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020887    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 00:17:55.022332 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020890    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 00:17:55.022332 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020894    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 00:17:55.022332 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020897    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 00:17:55.022332 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020900    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 00:17:55.022332 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020902    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 00:17:55.022332 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020906    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 00:17:55.022332 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020908    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 00:17:55.022332 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020911    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 00:17:55.022332 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020914    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 00:17:55.022835 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020917    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 00:17:55.022835 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020921    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 00:17:55.022835 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020923    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 00:17:55.022835 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020927    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 00:17:55.022835 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020930    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 00:17:55.022835 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020933    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 00:17:55.022835 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020937    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 00:17:55.022835 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020940    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 00:17:55.022835 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020942    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 00:17:55.022835 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020945    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 00:17:55.022835 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020950    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 00:17:55.022835 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020953    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 00:17:55.022835 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020958    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 00:17:55.022835 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020961    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 00:17:55.022835 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020965    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 00:17:55.022835 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020968    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 00:17:55.022835 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020970    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 00:17:55.022835 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.020974    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 00:17:55.022835 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021787    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 00:17:55.023354 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021796    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 00:17:55.023354 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021800    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 00:17:55.023354 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021804    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 00:17:55.023354 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021807    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 00:17:55.023354 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021810    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 00:17:55.023354 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021813    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 00:17:55.023354 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021816    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 00:17:55.023354 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021819    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 00:17:55.023354 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021822    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 00:17:55.023354 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021825    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 00:17:55.023354 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021828    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 00:17:55.023354 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021831    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 00:17:55.023354 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021834    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 00:17:55.023354 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021838    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 00:17:55.023354 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021842    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 00:17:55.023354 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021846    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 00:17:55.023354 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021849    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 00:17:55.023354 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021853    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 00:17:55.023354 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021857    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 00:17:55.023871 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021860    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 00:17:55.023871 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021863    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 00:17:55.023871 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021866    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 00:17:55.023871 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021869    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 00:17:55.023871 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021873    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 00:17:55.023871 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021877    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 00:17:55.023871 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021880    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 00:17:55.023871 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021883    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 00:17:55.023871 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021885    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 00:17:55.023871 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021888    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 00:17:55.023871 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021890    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 00:17:55.023871 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021893    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 00:17:55.023871 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021896    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 00:17:55.023871 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021899    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 00:17:55.023871 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021903    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 00:17:55.023871 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021907    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 00:17:55.023871 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021910    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 00:17:55.023871 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021913    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 00:17:55.023871 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021916    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021919    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021921    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021924    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021926    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021929    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021932    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021934    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021937    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021940    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021943    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021946    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021948    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021952    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021956    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021959    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021962    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021966    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021969    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021972    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 00:17:55.024394 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021975    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021978    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021981    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021984    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021987    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021989    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021992    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021995    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.021999    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022001    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022004    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022007    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022010    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022012    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022015    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022018    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022021    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022024    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022027    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022030    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 00:17:55.024963 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022032    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022035    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022038    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022041    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022044    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022047    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022050    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022053    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022056    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022059    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022062    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022065    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022068    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022071    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022074    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022076    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022079    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022082    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022085    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022087    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 00:17:55.025513 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022090    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022537    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022543    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022546    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022550    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022553    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022555    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022558    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022561    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022564    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022567    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022569    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022572    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022578    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022581    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022584    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022587    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022591    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022594    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022598    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 00:17:55.026043 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022601    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022603    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022606    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022610    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022612    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022615    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022618    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022621    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022624    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022626    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022629    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022632    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022635    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022637    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022640    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022643    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022646    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022649    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022651    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022654    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 00:17:55.026621 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022657    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 00:17:55.027137 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022660    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 00:17:55.027137 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022663    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 00:17:55.027137 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022665    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 00:17:55.027137 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022668    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 00:17:55.027137 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022673    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 00:17:55.027137 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022675    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 00:17:55.027137 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022678    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 00:17:55.027137 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022681    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 00:17:55.027137 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022684    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 00:17:55.027137 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022686    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 00:17:55.027137 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022689    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 00:17:55.027137 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022692    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 00:17:55.027137 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022695    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 00:17:55.027137 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022698    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 00:17:55.027137 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022701    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 00:17:55.027137 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022706    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 00:17:55.027137 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022710    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 00:17:55.027137 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022713    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 00:17:55.027137 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022716    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022719    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022722    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022724    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022728    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022731    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022734    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022737    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022740    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022742    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022745    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022748    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022751    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022753    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022756    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022759    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022761    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022764    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022768    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022771    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 00:17:55.027660 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022774    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 00:17:55.028221 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022777    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 00:17:55.028221 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022780    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 00:17:55.028221 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022783    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 00:17:55.028221 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022785    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 00:17:55.028221 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022790    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 00:17:55.028221 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022794    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 00:17:55.028221 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022797    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 00:17:55.028221 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022799    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 00:17:55.028221 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022803    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 00:17:55.028221 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022807    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 00:17:55.028221 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022813    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 00:17:55.028221 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022816    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 00:17:55.028221 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022819    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 00:17:55.028221 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022822    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 00:17:55.028221 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022825    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 00:17:55.028221 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022827    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 00:17:55.028221 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022830    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 00:17:55.028221 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022833    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 00:17:55.028221 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022836    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.022839    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022909    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022917    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022923    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022928    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022933    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022937    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022941    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022945    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022949    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022953    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022957    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022961    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022964    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022967    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022970    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022973    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022976    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022980    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022984    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022988    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022991    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022994    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 30 00:17:55.028748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.022999    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023003    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023006    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023009    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023013    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023016    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023020    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023023    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023027    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023030    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023034    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023037    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023040    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023043    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023047    2579 flags.go:64] FLAG: --enable-server="true"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023050    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023054    2579 flags.go:64] FLAG: --event-burst="100"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023058    2579 flags.go:64] FLAG: --event-qps="50"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023061    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023065    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023068    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023072    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023075    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023078    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023081    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 30 00:17:55.029395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023084    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023088    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023091    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023095    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023098    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023102    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023104    2579 flags.go:64] FLAG: --feature-gates=""
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023109    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023112    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023115    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023118    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023122    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023125    2579 flags.go:64] FLAG: --help="false"
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023128    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-135-68.ec2.internal"
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023131    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023134    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023137    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023141    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023144    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023148    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023151    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023154    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023157    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 30 00:17:55.030055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023161    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023164    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023168    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023171    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023174    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023177    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023180    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023183    2579 flags.go:64] FLAG: --lock-file=""
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023186    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023189    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023192    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023198    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023201    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023205    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023208    2579 flags.go:64] FLAG: --logging-format="text"
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023211    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023215    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023218    2579 flags.go:64] FLAG: --manifest-url=""
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023221    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023226    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023229    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023233    2579 flags.go:64] FLAG: --max-pods="110"
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023236    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023239    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023243    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 30 00:17:55.030722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023246    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023249    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023253    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023256    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023263    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023266    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023270    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023273    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023276    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023282    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023285    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023288    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023292    2579 flags.go:64] FLAG: --port="10250"
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023295    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023298    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06006a6737cbfd44b"
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023301    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023305    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023308    2579 flags.go:64] FLAG: --register-node="true"
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023311    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023327    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023331    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023335    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023338    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023343    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023346    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 30 00:17:55.031678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023349    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023352    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023355    2579 flags.go:64] FLAG: --runonce="false"
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023358    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023362    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023365    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023368    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023372    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023375    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023378    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023381    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023384    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023387    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023393    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023396    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023400    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023403    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023406    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023412    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023415    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023418    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023422    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023424    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023427    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023430    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 30 00:17:55.032652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023433    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023436    2579 flags.go:64] FLAG: --v="2"
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023440    2579 flags.go:64] FLAG: --version="false"
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023445    2579 flags.go:64] FLAG: --vmodule=""
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023449    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.023455    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023584    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023588    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023593    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023597    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023601    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023605    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023609    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023613    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023616    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023619    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023622    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023625    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023629    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023632    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 00:17:55.033450 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023636    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 00:17:55.034002 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023639    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 00:17:55.034002 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023642    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 00:17:55.034002 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023644    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 00:17:55.034002 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023647    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 00:17:55.034002 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023650    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 00:17:55.034002 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023653    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 00:17:55.034002 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023656    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 00:17:55.034002 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023659    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 00:17:55.034002 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023661    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 00:17:55.034002 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023664    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 00:17:55.034002 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023667    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 00:17:55.034002 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023671    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 00:17:55.034002 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023675    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 00:17:55.034002 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023678    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 00:17:55.034002 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023681    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 00:17:55.034002 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023684    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 00:17:55.034002 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023690    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 00:17:55.034002 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023693    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 00:17:55.034002 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023697    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023700    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023702    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023705    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023708    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023711    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023714    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023716    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023719    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023722    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023725    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023727    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023730    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023734    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023737    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023740    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023743    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023746    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023749    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023751    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 00:17:55.034515 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023754    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023757    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023760    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023762    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023765    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023768    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023772    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023775    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023777    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023780    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023786    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023789    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023792    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023795    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023798    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023800    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023803    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023806    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023809    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023812    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 00:17:55.035052 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023814    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023817    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023820    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023823    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023825    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023829    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023832    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023835    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023838    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023841    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023844    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023846    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023849    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023852    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023855    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023857    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023860    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023863    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023866    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023869    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 00:17:55.035589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023872    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 00:17:55.036124 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023875    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 00:17:55.036124 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023879    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 00:17:55.036124 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.023882    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 00:17:55.036124 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.024283    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 00:17:55.036124 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.024288    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 00:17:55.036124 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.024919    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 00:17:55.036124 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.034003    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 30 00:17:55.036124 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.034024    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 30 00:17:55.036124 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.034051    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 30 00:17:55.036124 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.034060    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 30 00:17:55.036124 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034148    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 00:17:55.036124 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034157    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 00:17:55.036124 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034163    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 00:17:55.036124 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034168    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 00:17:55.036124 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034172    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034176    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034181    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034185    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034190    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034194    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034199    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034205    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034210    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034215    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034221    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034226    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034231    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034236    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034241    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034246    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034251    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034256    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034261    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034266    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 00:17:55.036545 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034271    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034278    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034286    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034292    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034297    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034302    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034307    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034311    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034335    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034340    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034344    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034349    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034353    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034357    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034360    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034363    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034367    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034370    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034373    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034376    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 00:17:55.037079 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034379    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034382    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034387    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034390    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034393    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034396    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034398    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034401    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034404    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034407    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034410    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034413    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034417    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034419    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034422    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034427    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034429    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034433    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034435    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034439    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 00:17:55.037628 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034441    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 00:17:55.038176 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034444    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 00:17:55.038176 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034447    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 00:17:55.038176 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034451    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 00:17:55.038176 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034453    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 00:17:55.038176 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034457    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 00:17:55.038176 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034460    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 00:17:55.038176 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034463    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 00:17:55.038176 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034466    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 00:17:55.038176 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034470    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 00:17:55.038176 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034474    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 00:17:55.038176 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034477    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 00:17:55.038176 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034481    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 00:17:55.038176 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034483    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 00:17:55.038176 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034486    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 00:17:55.038176 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034490    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 00:17:55.038176 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034493    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 00:17:55.038176 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034495    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 00:17:55.038176 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034498    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 00:17:55.038176 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034501    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 00:17:55.038784 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034504    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 00:17:55.038784 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034507    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 00:17:55.038784 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034511    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 00:17:55.038784 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034515    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 00:17:55.038784 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034519    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 00:17:55.038784 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034523    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 00:17:55.038784 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034526    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 00:17:55.038784 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034529    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 00:17:55.038784 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034533    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 00:17:55.038784 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034536    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 00:17:55.038784 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034539    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 00:17:55.038784 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034542    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 00:17:55.038784 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034545    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 00:17:55.038784 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034548    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 00:17:55.038784 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034551    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 00:17:55.039167 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.034557    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 00:17:55.039167 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034686    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 00:17:55.039167 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034692    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 00:17:55.039167 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034697    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 00:17:55.039167 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034701    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 00:17:55.039167 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034705    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 00:17:55.039167 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034708    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 00:17:55.039167 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034711    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 00:17:55.039167 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034714    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 00:17:55.039167 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034717    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 00:17:55.039167 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034720    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 00:17:55.039167 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034723    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 00:17:55.039167 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034726    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 00:17:55.039167 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034730    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034732    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034735    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034738    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034741    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034744    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034747    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034750    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034753    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034756    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034759    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034762    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034764    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034767    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034770    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034772    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034775    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034778    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034780    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034783    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 00:17:55.039548 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034786    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 00:17:55.040063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034789    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 00:17:55.040063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034791    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 00:17:55.040063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034794    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 00:17:55.040063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034797    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 00:17:55.040063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034800    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 00:17:55.040063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034803    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 00:17:55.040063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034805    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 00:17:55.040063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034808    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 00:17:55.040063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034811    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 00:17:55.040063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034813    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 00:17:55.040063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034816    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 00:17:55.040063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034820    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 00:17:55.040063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034824    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 00:17:55.040063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034827    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 00:17:55.040063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034832    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 00:17:55.040063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034835    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 00:17:55.040063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034838    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 00:17:55.040063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034842    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 00:17:55.040063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034845    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034848    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034851    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034854    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034857    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034860    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034863    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034866    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034868    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034871    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034874    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034877    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034879    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034882    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034885    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034887    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034890    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034893    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034896    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034899    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 00:17:55.040593 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034901    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034904    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034907    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034909    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034912    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034915    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034917    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034922    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034925    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034927    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034930    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034933    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034936    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034938    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034941    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034945    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034948    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034951    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034976    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034981    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 00:17:55.041120 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034984    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 00:17:55.041758 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034988    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 00:17:55.041758 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034991    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 00:17:55.041758 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034994    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 00:17:55.041758 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.034997    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 00:17:55.041758 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.035000    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 00:17:55.041758 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.035002    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 00:17:55.041758 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:55.035005    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 00:17:55.041758 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.035010    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 00:17:55.041758 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.035132    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 30 00:17:55.041758 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.037434    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 30 00:17:55.041758 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.039435    2579 server.go:1019] "Starting client certificate rotation"
Jul 30 00:17:55.041758 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.039539    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 30 00:17:55.041758 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.039572    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 30 00:17:55.042313 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.042298    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 30 00:17:55.043599 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.043582    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 30 00:17:55.055302 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.055283    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 30 00:17:55.057875 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.057859    2579 log.go:25] "Validated CRI v1 image API"
Jul 30 00:17:55.058842 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.058826    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 30 00:17:55.061653 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.061632    2579 fs.go:135] Filesystem UUIDs: map[3e1f759f-c912-4ae9-8308-d31d74c75fc8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7b199999-f064-4ed2-8193-76d819d686a5:/dev/nvme0n1p4]
Jul 30 00:17:55.061711 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.061653    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 30 00:17:55.066994 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.066961    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 30 00:17:55.070582 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.070467    2579 manager.go:216] Machine: {Timestamp:2026-07-30 00:17:55.068926272 +0000 UTC m=+0.366193199 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102110 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dcbbb52ef0a5daee00f40a43b4bb4 SystemUUID:ec2dcbbb-52ef-0a5d-aee0-0f40a43b4bb4 BootID:74560bae-19ba-4748-9fff-0861d01fe01d Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:55:7d:ab:a7:fb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:55:7d:ab:a7:fb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:ce:7a:8d:b7:13 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 30 00:17:55.070582 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.070579    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 30 00:17:55.070693 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.070666    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 30 00:17:55.074847 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.074820    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 30 00:17:55.075029 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.074851    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-68.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 30 00:17:55.076190 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.076178    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 30 00:17:55.076226 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.076192    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 30 00:17:55.076226 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.076201    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 30 00:17:55.076984 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.076972    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 30 00:17:55.077851 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.077841    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 30 00:17:55.079987 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.079975    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 30 00:17:55.080104 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.080095    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 30 00:17:55.080142 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.080122    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 30 00:17:55.080142 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.080132    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 30 00:17:55.080199 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.080148    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 30 00:17:55.080199 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.080163    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 30 00:17:55.080199 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.080172    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 30 00:17:55.080615 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.080601    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 30 00:17:55.080672 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.080622    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 30 00:17:55.082962 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.082944    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 30 00:17:55.084963 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.084949    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 30 00:17:55.085007 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.084981    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 30 00:17:55.087256 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.087244    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 30 00:17:55.087299 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.087262    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 30 00:17:55.087299 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.087269    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 30 00:17:55.087299 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.087275    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 30 00:17:55.087299 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.087285    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 30 00:17:55.087299 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.087294    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 30 00:17:55.087464 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.087307    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 30 00:17:55.087464 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.087330    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 30 00:17:55.087464 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.087349    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 30 00:17:55.087464 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.087366    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 30 00:17:55.087464 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.087393    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 30 00:17:55.087464 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.087399    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 30 00:17:55.088745 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.088730    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 30 00:17:55.088794 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.088748    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 30 00:17:55.090767 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.090742    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 30 00:17:55.090826 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.090752    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-68.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 30 00:17:55.091232 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.091212    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7gkcr"
Jul 30 00:17:55.092461 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.092447    2579 server.go:1268] "Started kubelet"
Jul 30 00:17:55.092536 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.092512    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 30 00:17:55.092658 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.092592    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 30 00:17:55.092713 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.092703    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 30 00:17:55.093459 ip-10-0-135-68 systemd[1]: Started Kubernetes Kubelet.
Jul 30 00:17:55.095639 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.095623    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 30 00:17:55.098185 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.098160    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 30 00:17:55.098981 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.098961    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-68.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 30 00:17:55.099757 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.099730    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7gkcr"
Jul 30 00:17:55.099867 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.098868    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-68.ec2.internal.18c6e92b714867d8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-68.ec2.internal,UID:ip-10-0-135-68.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-68.ec2.internal,},FirstTimestamp:2026-07-30 00:17:55.092416472 +0000 UTC m=+0.389683396,LastTimestamp:2026-07-30 00:17:55.092416472 +0000 UTC m=+0.389683396,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-68.ec2.internal,}"
Jul 30 00:17:55.100368 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.100349    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 30 00:17:55.108112 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.108092    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 30 00:17:55.108195 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.108106    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 30 00:17:55.108485 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.108450    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-68.ec2.internal\" not found"
Jul 30 00:17:55.108868 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.108848    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 30 00:17:55.108868 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.108869    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 30 00:17:55.109006 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.108976    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 30 00:17:55.109006 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.108989    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 30 00:17:55.109136 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.109117    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 30 00:17:55.110618 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.110595    2579 factory.go:153] Registering CRI-O factory
Jul 30 00:17:55.110742 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.110637    2579 factory.go:223] Registration of the crio container factory successfully
Jul 30 00:17:55.110742 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.110693    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 30 00:17:55.110742 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.110727    2579 factory.go:55] Registering systemd factory
Jul 30 00:17:55.110742 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.110735    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 30 00:17:55.110939 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.110757    2579 factory.go:103] Registering Raw factory
Jul 30 00:17:55.110939 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.110779    2579 manager.go:1196] Started watching for new ooms in manager
Jul 30 00:17:55.111532 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.111515    2579 manager.go:349] Starting recovery of all containers
Jul 30 00:17:55.112773 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.112744    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 30 00:17:55.112867 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.112823    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-68.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 30 00:17:55.124038 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.124014    2579 manager.go:354] Recovery completed
Jul 30 00:17:55.128696 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.128683    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 00:17:55.130592 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.130569    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 00:17:55.130690 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.130607    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 00:17:55.130690 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.130623    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientPID"
Jul 30 00:17:55.131221 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.131204    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 30 00:17:55.131221 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.131217    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 00:17:55.131221 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.131220    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 30 00:17:55.131381 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.131241    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 30 00:17:55.131962 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.131948    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 00:17:55.132017 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.131976    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 00:17:55.132017 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.131988    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientPID"
Jul 30 00:17:55.132904 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.132891    2579 policy_none.go:49] "None policy: Start"
Jul 30 00:17:55.132961 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.132909    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 30 00:17:55.132961 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.132921    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 30 00:17:55.134185 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.134174    2579 policy_none.go:47] "Start"
Jul 30 00:17:55.175099 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.174926    2579 manager.go:337] "Starting Device Plugin manager"
Jul 30 00:17:55.175204 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.175125    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 30 00:17:55.175204 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.175143    2579 server.go:87] "Starting device plugin registration server"
Jul 30 00:17:55.175428 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.175414    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 30 00:17:55.175461 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.175432    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 30 00:17:55.175538 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.175520    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 30 00:17:55.175625 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.175616    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 30 00:17:55.175677 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.175627    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 30 00:17:55.176221 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.176202    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 30 00:17:55.176304 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.176257    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-68.ec2.internal\" not found"
Jul 30 00:17:55.275844 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.275772    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 00:17:55.276995 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.276788    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 00:17:55.276995 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.276822    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 00:17:55.276995 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.276834    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientPID"
Jul 30 00:17:55.276995 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.276877    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-68.ec2.internal"
Jul 30 00:17:55.285582 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.285559    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-68.ec2.internal"
Jul 30 00:17:55.285647 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.285593    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-68.ec2.internal\": node \"ip-10-0-135-68.ec2.internal\" not found"
Jul 30 00:17:55.303338 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.303316    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-68.ec2.internal\" not found"
Jul 30 00:17:55.314374 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.314341    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 30 00:17:55.315840 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.315815    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 30 00:17:55.315945 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.315864    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 30 00:17:55.315945 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.315892    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 30 00:17:55.315945 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.315936    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 30 00:17:55.318574 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.318556    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 00:17:55.406967 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.406926    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-68.ec2.internal\" not found"
Jul 30 00:17:55.417099 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.417078    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal"]
Jul 30 00:17:55.417157 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.417143    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 00:17:55.418771 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.418753    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 00:17:55.418834 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.418784    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 00:17:55.418834 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.418797    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientPID"
Jul 30 00:17:55.420728 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.420705    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 00:17:55.420993 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.420951    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal"
Jul 30 00:17:55.421342 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.421326    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 00:17:55.422127 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.422100    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 00:17:55.422127 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.422125    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 00:17:55.422245 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.422136    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientPID"
Jul 30 00:17:55.422245 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.422189    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 00:17:55.422245 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.422207    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 00:17:55.422245 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.422225    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientPID"
Jul 30 00:17:55.423428 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.423413    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal"
Jul 30 00:17:55.423489 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.423441    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 00:17:55.424158 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.424143    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 00:17:55.424270 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.424172    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 00:17:55.424270 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.424187    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientPID"
Jul 30 00:17:55.448068 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.448045    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-68.ec2.internal\" not found" node="ip-10-0-135-68.ec2.internal"
Jul 30 00:17:55.452850 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.452824    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-68.ec2.internal\" not found" node="ip-10-0-135-68.ec2.internal"
Jul 30 00:17:55.507389 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.507363    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-68.ec2.internal\" not found"
Jul 30 00:17:55.510935 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.510912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45dcbd02e4d8cba07fa3d3a342359b2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" (UID: \"45dcbd02e4d8cba07fa3d3a342359b2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal"
Jul 30 00:17:55.511017 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.510945    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/260636c800f3b5abe8a8cc948b2e682e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-68.ec2.internal\" (UID: \"260636c800f3b5abe8a8cc948b2e682e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal"
Jul 30 00:17:55.511017 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.510963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45dcbd02e4d8cba07fa3d3a342359b2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" (UID: \"45dcbd02e4d8cba07fa3d3a342359b2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal"
Jul 30 00:17:55.609488 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.609414    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-68.ec2.internal\" not found"
Jul 30 00:17:55.611777 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.611759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45dcbd02e4d8cba07fa3d3a342359b2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" (UID: \"45dcbd02e4d8cba07fa3d3a342359b2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal"
Jul 30 00:17:55.611831 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.611788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/260636c800f3b5abe8a8cc948b2e682e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-68.ec2.internal\" (UID: \"260636c800f3b5abe8a8cc948b2e682e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal"
Jul 30 00:17:55.611831 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.611809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45dcbd02e4d8cba07fa3d3a342359b2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" (UID: \"45dcbd02e4d8cba07fa3d3a342359b2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal"
Jul 30 00:17:55.611912 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.611861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/260636c800f3b5abe8a8cc948b2e682e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-68.ec2.internal\" (UID: \"260636c800f3b5abe8a8cc948b2e682e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal"
Jul 30 00:17:55.611912 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.611862    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45dcbd02e4d8cba07fa3d3a342359b2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" (UID: \"45dcbd02e4d8cba07fa3d3a342359b2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal"
Jul 30 00:17:55.611912 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.611896    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45dcbd02e4d8cba07fa3d3a342359b2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" (UID: \"45dcbd02e4d8cba07fa3d3a342359b2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal"
Jul 30 00:17:55.715919 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.715888    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-68.ec2.internal\" not found"
Jul 30 00:17:55.752857 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.752834    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal"
Jul 30 00:17:55.757159 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:55.757141    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal"
Jul 30 00:17:55.816272 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.816228    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-68.ec2.internal\" not found"
Jul 30 00:17:55.916443 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:55.916349    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-68.ec2.internal\" not found"
Jul 30 00:17:56.016421 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:56.016400    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-68.ec2.internal\" not found"
Jul 30 00:17:56.038838 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:56.038816    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 30 00:17:56.038994 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:56.038971    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 30 00:17:56.071388 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:56.071359    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 00:17:56.104753 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:56.104697    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-29 00:12:55 +0000 UTC" deadline="2028-02-20 10:08:38.790836583 +0000 UTC"
Jul 30 00:17:56.104753 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:56.104751    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13689h50m42.686090049s"
Jul 30 00:17:56.116853 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:56.116833    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-68.ec2.internal\" not found"
Jul 30 00:17:56.216980 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:56.216905    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-68.ec2.internal\" not found"
Jul 30 00:17:56.317480 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:56.317452    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-68.ec2.internal\" not found"
Jul 30 00:17:56.418465 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:56.418436    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-68.ec2.internal\" not found"
Jul 30 00:17:56.464444 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:56.464414    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 00:17:56.498329 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:56.498292    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260636c800f3b5abe8a8cc948b2e682e.slice/crio-306a69acf357cdcd5ab3b7e90f1cba16bb8498e23c824bf635c0369d6558f045 WatchSource:0}: Error finding container 306a69acf357cdcd5ab3b7e90f1cba16bb8498e23c824bf635c0369d6558f045: Status 404 returned error can't find the container with id 306a69acf357cdcd5ab3b7e90f1cba16bb8498e23c824bf635c0369d6558f045
Jul 30 00:17:56.504140 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:56.504120    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 00:17:56.519006 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:56.518981    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-68.ec2.internal\" not found"
Jul 30 00:17:56.609849 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:56.609822    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 00:17:56.714583 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:56.714553    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal"
Jul 30 00:17:56.726446 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:56.726422    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 30 00:17:56.727641 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:56.727628    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal"
Jul 30 00:17:56.734580 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:56.734565    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 30 00:17:57.060589 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:57.060554    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45dcbd02e4d8cba07fa3d3a342359b2c.slice/crio-ebbce9d07cbd96baae02f913c215417a440bc17075476e1551661e70e64019b3 WatchSource:0}: Error finding container ebbce9d07cbd96baae02f913c215417a440bc17075476e1551661e70e64019b3: Status 404 returned error can't find the container with id ebbce9d07cbd96baae02f913c215417a440bc17075476e1551661e70e64019b3
Jul 30 00:17:57.081423 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.081400    2579 apiserver.go:52] "Watching apiserver"
Jul 30 00:17:57.091684 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.091657    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 30 00:17:57.092322 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.092286    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-rglkm","openshift-image-registry/node-ca-pwx6g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal","openshift-multus/multus-9bqqn","openshift-multus/multus-additional-cni-plugins-jfkf9","openshift-multus/network-metrics-daemon-b6ns2","kube-system/konnectivity-agent-qbkcd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm","openshift-dns/node-resolver-2jclk","openshift-network-diagnostics/network-check-target-hg9rq","openshift-network-operator/iptables-alerter-cwm68","openshift-ovn-kubernetes/ovnkube-node-slhpw","kube-system/global-pull-secret-syncer-86f4q","kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal"]
Jul 30 00:17:57.094132 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.094103    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qbkcd"
Jul 30 00:17:57.095607 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.095585    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:17:57.095717 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:57.095662    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86f4q" podUID="ebd8b209-a805-4976-bb8c-24cba1f01d86"
Jul 30 00:17:57.097152 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.097128    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4t5zd\""
Jul 30 00:17:57.097409 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.097387    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 30 00:17:57.097626 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.097581    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 30 00:17:57.097852 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.097829    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.098811 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.098790    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.100143 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.100114    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.100923 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.100899    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 30 00:17:57.101034 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.100987    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mnfll\""
Jul 30 00:17:57.101518 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.101197    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 30 00:17:57.101518 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.101359    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 30 00:17:57.102235 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.102215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:17:57.102365 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:57.102307    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6ns2" podUID="a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1"
Jul 30 00:17:57.102825 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.102803    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 30 00:17:57.103114 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.103097    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zf875\""
Jul 30 00:17:57.104204 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.104174    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rz2tc\""
Jul 30 00:17:57.104382 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.104361    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 30 00:17:57.104854 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.104836    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 30 00:17:57.104928 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.104872    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 30 00:17:57.105518 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.105495    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 30 00:17:57.105879 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.105863    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 30 00:17:57.107947 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.107842    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2jclk"
Jul 30 00:17:57.109958 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.109464    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 30 00:17:57.110185 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.110162    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.110696 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.110641    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 30 00:17:57.110935 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.110651    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4jzh5\""
Jul 30 00:17:57.111050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.110757    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 30 00:17:57.111853 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.111837    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pwx6g"
Jul 30 00:17:57.113391 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.113371    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 30 00:17:57.113886 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.113608    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-stk5q\""
Jul 30 00:17:57.113886 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.113676    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 30 00:17:57.113886 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.113888    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:17:57.114166 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:57.113951    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hg9rq" podUID="3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0"
Jul 30 00:17:57.114542 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.114508    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 30 00:17:57.114710 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.114689    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kgd84\""
Jul 30 00:17:57.114920 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.114903    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 30 00:17:57.115009 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.114942    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 30 00:17:57.118397 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.115610    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cwm68"
Jul 30 00:17:57.118397 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.117612    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.118563 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.118510    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 30 00:17:57.119269 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.118759    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 30 00:17:57.119269 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.118792    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 30 00:17:57.119269 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.119082    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2zz5r\""
Jul 30 00:17:57.120649 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.120291    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 30 00:17:57.120935 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.120919    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 30 00:17:57.121259 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.121053    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 30 00:17:57.121538 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.121518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f6d919d-802a-4eb8-a7ec-948626dfc26b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.121677 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.121554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ebd8b209-a805-4976-bb8c-24cba1f01d86-dbus\") pod \"global-pull-secret-syncer-86f4q\" (UID: \"ebd8b209-a805-4976-bb8c-24cba1f01d86\") " pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:17:57.121677 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.121581    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-host-run-multus-certs\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.121677 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.121606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-host-var-lib-cni-bin\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.121677 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.121634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q648l\" (UniqueName: \"kubernetes.io/projected/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-kube-api-access-q648l\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.121901 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.121683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-registration-dir\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.121901 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.121735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/182ce8f5-d95e-479e-9adf-a5571616b663-konnectivity-ca\") pod \"konnectivity-agent-qbkcd\" (UID: \"182ce8f5-d95e-479e-9adf-a5571616b663\") " pod="kube-system/konnectivity-agent-qbkcd"
Jul 30 00:17:57.121901 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.121766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.121901 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.121797    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-sysctl-conf\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.121901 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.121836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f6d919d-802a-4eb8-a7ec-948626dfc26b-cnibin\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.122173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.121907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f6d919d-802a-4eb8-a7ec-948626dfc26b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.122173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.121934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-system-cni-dir\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.122173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-hostroot\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.122173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122049    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 30 00:17:57.122173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-multus-conf-dir\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.122173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122108    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 30 00:17:57.122173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-multus-daemon-config\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.122173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122169    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 30 00:17:57.122592 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpkc6\" (UniqueName: \"kubernetes.io/projected/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-kube-api-access-qpkc6\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.122592 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f6d919d-802a-4eb8-a7ec-948626dfc26b-system-cni-dir\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.122592 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/318afb78-6a56-4524-8c63-8a3c2b223271-tmp-dir\") pod \"node-resolver-2jclk\" (UID: \"318afb78-6a56-4524-8c63-8a3c2b223271\") " pod="openshift-dns/node-resolver-2jclk"
Jul 30 00:17:57.122592 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-sysconfig\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.122592 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-lib-modules\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.122592 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-host-run-k8s-cni-cncf-io\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.122592 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-host-var-lib-cni-multus\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.122592 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-systemd\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.122592 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122478    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 30 00:17:57.122592 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-run\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.122592 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zhn8\" (UniqueName: \"kubernetes.io/projected/7f6d919d-802a-4eb8-a7ec-948626dfc26b-kube-api-access-2zhn8\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.122592 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csv4t\" (UniqueName: \"kubernetes.io/projected/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-kube-api-access-csv4t\") pod \"network-metrics-daemon-b6ns2\" (UID: \"a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1\") " pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:17:57.123173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-host-run-netns\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.123173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122618    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bdwfx\""
Jul 30 00:17:57.123173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwhrr\" (UniqueName: \"kubernetes.io/projected/836811a6-7584-4818-884a-59ce980afe22-kube-api-access-lwhrr\") pod \"node-ca-pwx6g\" (UID: \"836811a6-7584-4818-884a-59ce980afe22\") " pod="openshift-image-registry/node-ca-pwx6g"
Jul 30 00:17:57.123173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122737    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kt7m\" (UniqueName: \"kubernetes.io/projected/318afb78-6a56-4524-8c63-8a3c2b223271-kube-api-access-8kt7m\") pod \"node-resolver-2jclk\" (UID: \"318afb78-6a56-4524-8c63-8a3c2b223271\") " pod="openshift-dns/node-resolver-2jclk"
Jul 30 00:17:57.123173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-modprobe-d\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.123173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122797    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/182ce8f5-d95e-479e-9adf-a5571616b663-agent-certs\") pod \"konnectivity-agent-qbkcd\" (UID: \"182ce8f5-d95e-479e-9adf-a5571616b663\") " pod="kube-system/konnectivity-agent-qbkcd"
Jul 30 00:17:57.123173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-cnibin\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.123173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122857    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-multus-socket-dir-parent\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.123173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-host-var-lib-kubelet\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.123173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-etc-selinux\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.123173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122940    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f6d919d-802a-4eb8-a7ec-948626dfc26b-cni-binary-copy\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.123173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.122965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/836811a6-7584-4818-884a-59ce980afe22-serviceca\") pod \"node-ca-pwx6g\" (UID: \"836811a6-7584-4818-884a-59ce980afe22\") " pod="openshift-image-registry/node-ca-pwx6g"
Jul 30 00:17:57.125111 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125085    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret\") pod \"global-pull-secret-syncer-86f4q\" (UID: \"ebd8b209-a805-4976-bb8c-24cba1f01d86\") " pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:17:57.125213 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-socket-dir\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.125213 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ebd8b209-a805-4976-bb8c-24cba1f01d86-kubelet-config\") pod \"global-pull-secret-syncer-86f4q\" (UID: \"ebd8b209-a805-4976-bb8c-24cba1f01d86\") " pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:17:57.125213 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-cni-binary-copy\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.125488 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/318afb78-6a56-4524-8c63-8a3c2b223271-hosts-file\") pod \"node-resolver-2jclk\" (UID: \"318afb78-6a56-4524-8c63-8a3c2b223271\") " pod="openshift-dns/node-resolver-2jclk"
Jul 30 00:17:57.125488 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f6d919d-802a-4eb8-a7ec-948626dfc26b-os-release\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.125488 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-os-release\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.125488 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-kubernetes\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.125488 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125394    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-sysctl-d\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.125488 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-var-lib-kubelet\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.125812 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1332083-d23f-44ac-a0f2-8d6046a0fe48-tmp\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.125812 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f6d919d-802a-4eb8-a7ec-948626dfc26b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.125812 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125595    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-device-dir\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.125812 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-sys-fs\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.125812 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/836811a6-7584-4818-884a-59ce980afe22-host\") pod \"node-ca-pwx6g\" (UID: \"836811a6-7584-4818-884a-59ce980afe22\") " pod="openshift-image-registry/node-ca-pwx6g"
Jul 30 00:17:57.125812 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-etc-kubernetes\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.125812 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-multus-cni-dir\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.126200 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-sys\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.126200 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-tuned\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.126200 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f4fw\" (UniqueName: \"kubernetes.io/projected/c1332083-d23f-44ac-a0f2-8d6046a0fe48-kube-api-access-2f4fw\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.126200 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs\") pod \"network-metrics-daemon-b6ns2\" (UID: \"a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1\") " pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:17:57.126200 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.125932    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-host\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.143922 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.143886    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vgh54"
Jul 30 00:17:57.150817 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.150794    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vgh54"
Jul 30 00:17:57.210726 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.210700    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 30 00:17:57.226389 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/836811a6-7584-4818-884a-59ce980afe22-host\") pod \"node-ca-pwx6g\" (UID: \"836811a6-7584-4818-884a-59ce980afe22\") " pod="openshift-image-registry/node-ca-pwx6g"
Jul 30 00:17:57.226389 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-etc-kubernetes\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.226709 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmm96\" (UniqueName: \"kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96\") pod \"network-check-target-hg9rq\" (UID: \"3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0\") " pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:17:57.226709 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226499    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-etc-kubernetes\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.226709 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226499    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/836811a6-7584-4818-884a-59ce980afe22-host\") pod \"node-ca-pwx6g\" (UID: \"836811a6-7584-4818-884a-59ce980afe22\") " pod="openshift-image-registry/node-ca-pwx6g"
Jul 30 00:17:57.226709 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-multus-cni-dir\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.226709 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-sys\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.226709 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-tuned\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.226709 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226661    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f4fw\" (UniqueName: \"kubernetes.io/projected/c1332083-d23f-44ac-a0f2-8d6046a0fe48-kube-api-access-2f4fw\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.226709 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs\") pod \"network-metrics-daemon-b6ns2\" (UID: \"a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1\") " pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:17:57.226709 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-host\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.227148 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226738    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-run-ovn\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.227148 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc610fab-9658-40eb-8527-081a0008d70f-ovnkube-script-lib\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.227148 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226783    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-sys\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.227148 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f6d919d-802a-4eb8-a7ec-948626dfc26b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.227148 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-multus-cni-dir\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.227148 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ebd8b209-a805-4976-bb8c-24cba1f01d86-dbus\") pod \"global-pull-secret-syncer-86f4q\" (UID: \"ebd8b209-a805-4976-bb8c-24cba1f01d86\") " pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:17:57.227148 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226837    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-host\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.227148 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226853    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-host-run-multus-certs\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.227148 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1856c95-b8d8-4303-9ec9-04f3f8f3dec2-host-slash\") pod \"iptables-alerter-cwm68\" (UID: \"c1856c95-b8d8-4303-9ec9-04f3f8f3dec2\") " pod="openshift-network-operator/iptables-alerter-cwm68"
Jul 30 00:17:57.227148 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:57.226901    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 00:17:57.227148 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226910    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-host-var-lib-cni-bin\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.227148 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226915    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-host-run-multus-certs\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.227148 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q648l\" (UniqueName: \"kubernetes.io/projected/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-kube-api-access-q648l\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.227148 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:57.226987    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs podName:a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1 nodeName:}" failed. No retries permitted until 2026-07-30 00:17:57.726943777 +0000 UTC m=+3.024210711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs") pod "network-metrics-daemon-b6ns2" (UID: "a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 00:17:57.227148 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-host-var-lib-cni-bin\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.227148 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.226974    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ebd8b209-a805-4976-bb8c-24cba1f01d86-dbus\") pod \"global-pull-secret-syncer-86f4q\" (UID: \"ebd8b209-a805-4976-bb8c-24cba1f01d86\") " pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:17:57.227148 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-registration-dir\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.227998 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-slash\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.227998 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227173    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-registration-dir\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.227998 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-log-socket\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.227998 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227341    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/182ce8f5-d95e-479e-9adf-a5571616b663-konnectivity-ca\") pod \"konnectivity-agent-qbkcd\" (UID: \"182ce8f5-d95e-479e-9adf-a5571616b663\") " pod="kube-system/konnectivity-agent-qbkcd"
Jul 30 00:17:57.227998 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.227998 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-sysctl-conf\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.227998 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227430    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-systemd-units\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.227998 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-cni-netd\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.227998 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f6d919d-802a-4eb8-a7ec-948626dfc26b-cnibin\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.227998 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227481    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 30 00:17:57.227998 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227530    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f6d919d-802a-4eb8-a7ec-948626dfc26b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.227998 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-system-cni-dir\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.227998 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-hostroot\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.227998 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-multus-conf-dir\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.227998 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-multus-daemon-config\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.227998 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpkc6\" (UniqueName: \"kubernetes.io/projected/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-kube-api-access-qpkc6\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.227998 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227696    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f6d919d-802a-4eb8-a7ec-948626dfc26b-system-cni-dir\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.229050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/318afb78-6a56-4524-8c63-8a3c2b223271-tmp-dir\") pod \"node-resolver-2jclk\" (UID: \"318afb78-6a56-4524-8c63-8a3c2b223271\") " pod="openshift-dns/node-resolver-2jclk"
Jul 30 00:17:57.229050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227742    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f6d919d-802a-4eb8-a7ec-948626dfc26b-cnibin\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.229050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-sysconfig\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.229050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227786    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f6d919d-802a-4eb8-a7ec-948626dfc26b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.229050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-lib-modules\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.229050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227811    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.229050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-multus-conf-dir\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.229050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-node-log\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.229050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227859    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/182ce8f5-d95e-479e-9adf-a5571616b663-konnectivity-ca\") pod \"konnectivity-agent-qbkcd\" (UID: \"182ce8f5-d95e-479e-9adf-a5571616b663\") " pod="kube-system/konnectivity-agent-qbkcd"
Jul 30 00:17:57.229050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227880    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f6d919d-802a-4eb8-a7ec-948626dfc26b-system-cni-dir\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.229050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227878    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-sysctl-conf\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.229050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.227960    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-lib-modules\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.229050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-hostroot\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.229050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f6d919d-802a-4eb8-a7ec-948626dfc26b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.229050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228015    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.229050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-host-run-k8s-cni-cncf-io\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.229050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-host-var-lib-cni-multus\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.230045 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-host-run-k8s-cni-cncf-io\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.230045 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-systemd\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.230045 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-run\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.230045 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-host-var-lib-cni-multus\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.230045 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-sysconfig\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.230045 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228073    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-system-cni-dir\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.230045 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-run-openvswitch\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.230045 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zhn8\" (UniqueName: \"kubernetes.io/projected/7f6d919d-802a-4eb8-a7ec-948626dfc26b-kube-api-access-2zhn8\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.230045 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228261    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csv4t\" (UniqueName: \"kubernetes.io/projected/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-kube-api-access-csv4t\") pod \"network-metrics-daemon-b6ns2\" (UID: \"a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1\") " pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:17:57.230045 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-run\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.230045 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/318afb78-6a56-4524-8c63-8a3c2b223271-tmp-dir\") pod \"node-resolver-2jclk\" (UID: \"318afb78-6a56-4524-8c63-8a3c2b223271\") " pod="openshift-dns/node-resolver-2jclk"
Jul 30 00:17:57.230045 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228343    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-host-run-netns\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.230045 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-host-run-netns\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.230045 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228408    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc610fab-9658-40eb-8527-081a0008d70f-ovnkube-config\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.230045 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-systemd\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.230045 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwhrr\" (UniqueName: \"kubernetes.io/projected/836811a6-7584-4818-884a-59ce980afe22-kube-api-access-lwhrr\") pod \"node-ca-pwx6g\" (UID: \"836811a6-7584-4818-884a-59ce980afe22\") " pod="openshift-image-registry/node-ca-pwx6g"
Jul 30 00:17:57.230045 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kt7m\" (UniqueName: \"kubernetes.io/projected/318afb78-6a56-4524-8c63-8a3c2b223271-kube-api-access-8kt7m\") pod \"node-resolver-2jclk\" (UID: \"318afb78-6a56-4524-8c63-8a3c2b223271\") " pod="openshift-dns/node-resolver-2jclk"
Jul 30 00:17:57.230045 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228476    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-multus-daemon-config\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.230917 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-modprobe-d\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.230917 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228548    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx7tg\" (UniqueName: \"kubernetes.io/projected/fc610fab-9658-40eb-8527-081a0008d70f-kube-api-access-tx7tg\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.230917 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/182ce8f5-d95e-479e-9adf-a5571616b663-agent-certs\") pod \"konnectivity-agent-qbkcd\" (UID: \"182ce8f5-d95e-479e-9adf-a5571616b663\") " pod="kube-system/konnectivity-agent-qbkcd"
Jul 30 00:17:57.230917 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-modprobe-d\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.230917 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-cnibin\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.230917 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-var-lib-openvswitch\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.230917 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228653    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-cnibin\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.230917 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-etc-openvswitch\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.230917 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-multus-socket-dir-parent\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.230917 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-host-var-lib-kubelet\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.230917 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228746    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-etc-selinux\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.230917 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-multus-socket-dir-parent\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.230917 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228789    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-host-var-lib-kubelet\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.230917 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f6d919d-802a-4eb8-a7ec-948626dfc26b-cni-binary-copy\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.230917 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/836811a6-7584-4818-884a-59ce980afe22-serviceca\") pod \"node-ca-pwx6g\" (UID: \"836811a6-7584-4818-884a-59ce980afe22\") " pod="openshift-image-registry/node-ca-pwx6g"
Jul 30 00:17:57.230917 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228849    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-etc-selinux\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.230917 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret\") pod \"global-pull-secret-syncer-86f4q\" (UID: \"ebd8b209-a805-4976-bb8c-24cba1f01d86\") " pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:17:57.231631 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-socket-dir\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.231631 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228926    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ebd8b209-a805-4976-bb8c-24cba1f01d86-kubelet-config\") pod \"global-pull-secret-syncer-86f4q\" (UID: \"ebd8b209-a805-4976-bb8c-24cba1f01d86\") " pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:17:57.231631 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:57.228963    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 00:17:57.231631 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.228967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-cni-binary-copy\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.231631 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/318afb78-6a56-4524-8c63-8a3c2b223271-hosts-file\") pod \"node-resolver-2jclk\" (UID: \"318afb78-6a56-4524-8c63-8a3c2b223271\") " pod="openshift-dns/node-resolver-2jclk"
Jul 30 00:17:57.231631 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:57.229026    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret podName:ebd8b209-a805-4976-bb8c-24cba1f01d86 nodeName:}" failed. No retries permitted until 2026-07-30 00:17:57.729007397 +0000 UTC m=+3.026274324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret") pod "global-pull-secret-syncer-86f4q" (UID: "ebd8b209-a805-4976-bb8c-24cba1f01d86") : object "kube-system"/"original-pull-secret" not registered
Jul 30 00:17:57.231631 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229047    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/318afb78-6a56-4524-8c63-8a3c2b223271-hosts-file\") pod \"node-resolver-2jclk\" (UID: \"318afb78-6a56-4524-8c63-8a3c2b223271\") " pod="openshift-dns/node-resolver-2jclk"
Jul 30 00:17:57.231631 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229048    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-socket-dir\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.231631 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zntk\" (UniqueName: \"kubernetes.io/projected/c1856c95-b8d8-4303-9ec9-04f3f8f3dec2-kube-api-access-8zntk\") pod \"iptables-alerter-cwm68\" (UID: \"c1856c95-b8d8-4303-9ec9-04f3f8f3dec2\") " pod="openshift-network-operator/iptables-alerter-cwm68"
Jul 30 00:17:57.231631 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-run-netns\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.231631 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ebd8b209-a805-4976-bb8c-24cba1f01d86-kubelet-config\") pod \"global-pull-secret-syncer-86f4q\" (UID: \"ebd8b209-a805-4976-bb8c-24cba1f01d86\") " pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:17:57.231631 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-run-systemd\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.231631 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc610fab-9658-40eb-8527-081a0008d70f-env-overrides\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.231631 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229281    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc610fab-9658-40eb-8527-081a0008d70f-ovn-node-metrics-cert\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.231631 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f6d919d-802a-4eb8-a7ec-948626dfc26b-os-release\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.231631 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-os-release\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.231631 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229348    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f6d919d-802a-4eb8-a7ec-948626dfc26b-cni-binary-copy\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.232370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-kubernetes\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.232370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-sysctl-d\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.232370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229401    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/836811a6-7584-4818-884a-59ce980afe22-serviceca\") pod \"node-ca-pwx6g\" (UID: \"836811a6-7584-4818-884a-59ce980afe22\") " pod="openshift-image-registry/node-ca-pwx6g"
Jul 30 00:17:57.232370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-var-lib-kubelet\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.232370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229441    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f6d919d-802a-4eb8-a7ec-948626dfc26b-os-release\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.232370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229446    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-os-release\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.232370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229505    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-kubernetes\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.232370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1332083-d23f-44ac-a0f2-8d6046a0fe48-tmp\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.232370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229508    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-var-lib-kubelet\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.232370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229522    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-sysctl-d\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.232370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1856c95-b8d8-4303-9ec9-04f3f8f3dec2-iptables-alerter-script\") pod \"iptables-alerter-cwm68\" (UID: \"c1856c95-b8d8-4303-9ec9-04f3f8f3dec2\") " pod="openshift-network-operator/iptables-alerter-cwm68"
Jul 30 00:17:57.232370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229570    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-kubelet\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.232370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-cni-binary-copy\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.232370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f6d919d-802a-4eb8-a7ec-948626dfc26b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.232370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-device-dir\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.232370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-run-ovn-kubernetes\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.232370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-cni-bin\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.232986 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-device-dir\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.232986 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-sys-fs\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.232986 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.229804    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-sys-fs\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.232986 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.230083    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f6d919d-802a-4eb8-a7ec-948626dfc26b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.232986 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.231725    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1332083-d23f-44ac-a0f2-8d6046a0fe48-etc-tuned\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.232986 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.231972    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1332083-d23f-44ac-a0f2-8d6046a0fe48-tmp\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.232986 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.232169    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/182ce8f5-d95e-479e-9adf-a5571616b663-agent-certs\") pod \"konnectivity-agent-qbkcd\" (UID: \"182ce8f5-d95e-479e-9adf-a5571616b663\") " pod="kube-system/konnectivity-agent-qbkcd"
Jul 30 00:17:57.236793 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.236763    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q648l\" (UniqueName: \"kubernetes.io/projected/8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0-kube-api-access-q648l\") pod \"aws-ebs-csi-driver-node-kgfxm\" (UID: \"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.236923 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.236845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f4fw\" (UniqueName: \"kubernetes.io/projected/c1332083-d23f-44ac-a0f2-8d6046a0fe48-kube-api-access-2f4fw\") pod \"tuned-rglkm\" (UID: \"c1332083-d23f-44ac-a0f2-8d6046a0fe48\") " pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.238414 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.238377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpkc6\" (UniqueName: \"kubernetes.io/projected/7471b635-17bb-4f3a-b41f-ee9fa0f9871e-kube-api-access-qpkc6\") pod \"multus-9bqqn\" (UID: \"7471b635-17bb-4f3a-b41f-ee9fa0f9871e\") " pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.238864 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.238841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwhrr\" (UniqueName: \"kubernetes.io/projected/836811a6-7584-4818-884a-59ce980afe22-kube-api-access-lwhrr\") pod \"node-ca-pwx6g\" (UID: \"836811a6-7584-4818-884a-59ce980afe22\") " pod="openshift-image-registry/node-ca-pwx6g"
Jul 30 00:17:57.239076 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.239058    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csv4t\" (UniqueName: \"kubernetes.io/projected/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-kube-api-access-csv4t\") pod \"network-metrics-daemon-b6ns2\" (UID: \"a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1\") " pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:17:57.239181 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.239162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zhn8\" (UniqueName: \"kubernetes.io/projected/7f6d919d-802a-4eb8-a7ec-948626dfc26b-kube-api-access-2zhn8\") pod \"multus-additional-cni-plugins-jfkf9\" (UID: \"7f6d919d-802a-4eb8-a7ec-948626dfc26b\") " pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.239283 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.239263    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kt7m\" (UniqueName: \"kubernetes.io/projected/318afb78-6a56-4524-8c63-8a3c2b223271-kube-api-access-8kt7m\") pod \"node-resolver-2jclk\" (UID: \"318afb78-6a56-4524-8c63-8a3c2b223271\") " pod="openshift-dns/node-resolver-2jclk"
Jul 30 00:17:57.321940 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.321789    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" event={"ID":"45dcbd02e4d8cba07fa3d3a342359b2c","Type":"ContainerStarted","Data":"ebbce9d07cbd96baae02f913c215417a440bc17075476e1551661e70e64019b3"}
Jul 30 00:17:57.323486 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.323425    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" event={"ID":"260636c800f3b5abe8a8cc948b2e682e","Type":"ContainerStarted","Data":"306a69acf357cdcd5ab3b7e90f1cba16bb8498e23c824bf635c0369d6558f045"}
Jul 30 00:17:57.330874 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.330842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-var-lib-openvswitch\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.330997 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.330882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-etc-openvswitch\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.330997 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.330934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zntk\" (UniqueName: \"kubernetes.io/projected/c1856c95-b8d8-4303-9ec9-04f3f8f3dec2-kube-api-access-8zntk\") pod \"iptables-alerter-cwm68\" (UID: \"c1856c95-b8d8-4303-9ec9-04f3f8f3dec2\") " pod="openshift-network-operator/iptables-alerter-cwm68"
Jul 30 00:17:57.330997 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.330962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-run-netns\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.330997 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.330986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-run-systemd\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.331195 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331011    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc610fab-9658-40eb-8527-081a0008d70f-env-overrides\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.331195 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc610fab-9658-40eb-8527-081a0008d70f-ovn-node-metrics-cert\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.331195 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-etc-openvswitch\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.331195 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1856c95-b8d8-4303-9ec9-04f3f8f3dec2-iptables-alerter-script\") pod \"iptables-alerter-cwm68\" (UID: \"c1856c95-b8d8-4303-9ec9-04f3f8f3dec2\") " pod="openshift-network-operator/iptables-alerter-cwm68"
Jul 30 00:17:57.331195 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-kubelet\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.331195 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-var-lib-openvswitch\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.331195 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-run-ovn-kubernetes\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.331195 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-cni-bin\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.331195 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmm96\" (UniqueName: \"kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96\") pod \"network-check-target-hg9rq\" (UID: \"3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0\") " pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:17:57.331663 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-run-ovn\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.331663 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc610fab-9658-40eb-8527-081a0008d70f-ovnkube-script-lib\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.331663 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1856c95-b8d8-4303-9ec9-04f3f8f3dec2-host-slash\") pod \"iptables-alerter-cwm68\" (UID: \"c1856c95-b8d8-4303-9ec9-04f3f8f3dec2\") " pod="openshift-network-operator/iptables-alerter-cwm68"
Jul 30 00:17:57.331663 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331308    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-slash\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.331663 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-log-socket\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.331663 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-systemd-units\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.331663 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-cni-netd\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.331663 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-node-log\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.331663 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.331663 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-run-openvswitch\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.331663 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc610fab-9658-40eb-8527-081a0008d70f-ovnkube-config\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.331663 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx7tg\" (UniqueName: \"kubernetes.io/projected/fc610fab-9658-40eb-8527-081a0008d70f-kube-api-access-tx7tg\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.331663 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331615    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-run-netns\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.332351 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1856c95-b8d8-4303-9ec9-04f3f8f3dec2-host-slash\") pod \"iptables-alerter-cwm68\" (UID: \"c1856c95-b8d8-4303-9ec9-04f3f8f3dec2\") " pod="openshift-network-operator/iptables-alerter-cwm68"
Jul 30 00:17:57.332351 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.331964    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-slash\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.332351 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.332000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-cni-bin\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.332351 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.332011    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-log-socket\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.332351 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.332063    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-kubelet\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.332351 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.332064    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-cni-netd\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.332351 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.332101    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-node-log\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.332351 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.332114    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1856c95-b8d8-4303-9ec9-04f3f8f3dec2-iptables-alerter-script\") pod \"iptables-alerter-cwm68\" (UID: \"c1856c95-b8d8-4303-9ec9-04f3f8f3dec2\") " pod="openshift-network-operator/iptables-alerter-cwm68"
Jul 30 00:17:57.332351 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.332117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-run-ovn-kubernetes\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.332351 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.332148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.332351 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.332151    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-systemd-units\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.332351 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.332187    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-run-ovn\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.332351 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.332233    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-run-openvswitch\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.332351 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.332256    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc610fab-9658-40eb-8527-081a0008d70f-run-systemd\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.333042 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.332667    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc610fab-9658-40eb-8527-081a0008d70f-env-overrides\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.333042 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.332828    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc610fab-9658-40eb-8527-081a0008d70f-ovnkube-config\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.333042 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.332875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc610fab-9658-40eb-8527-081a0008d70f-ovnkube-script-lib\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.334609 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.334573    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc610fab-9658-40eb-8527-081a0008d70f-ovn-node-metrics-cert\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.338293 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:57.338229    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 00:17:57.338293 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:57.338256    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 00:17:57.338293 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:57.338274    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bmm96 for pod openshift-network-diagnostics/network-check-target-hg9rq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 00:17:57.338528 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:57.338357    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96 podName:3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0 nodeName:}" failed. No retries permitted until 2026-07-30 00:17:57.838320829 +0000 UTC m=+3.135587761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bmm96" (UniqueName: "kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96") pod "network-check-target-hg9rq" (UID: "3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 00:17:57.341282 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.341233    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zntk\" (UniqueName: \"kubernetes.io/projected/c1856c95-b8d8-4303-9ec9-04f3f8f3dec2-kube-api-access-8zntk\") pod \"iptables-alerter-cwm68\" (UID: \"c1856c95-b8d8-4303-9ec9-04f3f8f3dec2\") " pod="openshift-network-operator/iptables-alerter-cwm68"
Jul 30 00:17:57.341406 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.341349    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx7tg\" (UniqueName: \"kubernetes.io/projected/fc610fab-9658-40eb-8527-081a0008d70f-kube-api-access-tx7tg\") pod \"ovnkube-node-slhpw\" (UID: \"fc610fab-9658-40eb-8527-081a0008d70f\") " pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.410933 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.410893    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qbkcd"
Jul 30 00:17:57.420423 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.420395    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm"
Jul 30 00:17:57.430358 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.430330    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9bqqn"
Jul 30 00:17:57.437276 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.437241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jfkf9"
Jul 30 00:17:57.446133 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.446078    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2jclk"
Jul 30 00:17:57.455194 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.455173    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rglkm"
Jul 30 00:17:57.462952 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.462932    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pwx6g"
Jul 30 00:17:57.472182 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.472158    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cwm68"
Jul 30 00:17:57.479423 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.479403    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:17:57.630961 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.630866    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 00:17:57.734058 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.734024    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs\") pod \"network-metrics-daemon-b6ns2\" (UID: \"a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1\") " pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:17:57.734228 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.734082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret\") pod \"global-pull-secret-syncer-86f4q\" (UID: \"ebd8b209-a805-4976-bb8c-24cba1f01d86\") " pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:17:57.734228 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:57.734184    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 00:17:57.734228 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:57.734195    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 00:17:57.734410 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:57.734265    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs podName:a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1 nodeName:}" failed. No retries permitted until 2026-07-30 00:17:58.734235686 +0000 UTC m=+4.031502601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs") pod "network-metrics-daemon-b6ns2" (UID: "a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 00:17:57.734410 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:57.734282    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret podName:ebd8b209-a805-4976-bb8c-24cba1f01d86 nodeName:}" failed. No retries permitted until 2026-07-30 00:17:58.734274851 +0000 UTC m=+4.031541768 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret") pod "global-pull-secret-syncer-86f4q" (UID: "ebd8b209-a805-4976-bb8c-24cba1f01d86") : object "kube-system"/"original-pull-secret" not registered
Jul 30 00:17:57.936215 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:57.936114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmm96\" (UniqueName: \"kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96\") pod \"network-check-target-hg9rq\" (UID: \"3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0\") " pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:17:57.936406 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:57.936332    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 00:17:57.936406 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:57.936358    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 00:17:57.936406 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:57.936373    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bmm96 for pod openshift-network-diagnostics/network-check-target-hg9rq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 00:17:57.936585 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:57.936448    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96 podName:3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0 nodeName:}" failed. No retries permitted until 2026-07-30 00:17:58.936426401 +0000 UTC m=+4.233693331 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bmm96" (UniqueName: "kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96") pod "network-check-target-hg9rq" (UID: "3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 00:17:58.134669 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:58.134635    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a72c231_04bd_4d8c_a0dd_c59d5a61c4a0.slice/crio-eec8c4cf48997e0889e57313d55dafc4623b15ba7a5602acb500baa66efea53d WatchSource:0}: Error finding container eec8c4cf48997e0889e57313d55dafc4623b15ba7a5602acb500baa66efea53d: Status 404 returned error can't find the container with id eec8c4cf48997e0889e57313d55dafc4623b15ba7a5602acb500baa66efea53d
Jul 30 00:17:58.139290 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:58.139261    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6d919d_802a_4eb8_a7ec_948626dfc26b.slice/crio-641c49944e551bf71c97262a300dff3433d0215f4775b55c01ecda24d49d9156 WatchSource:0}: Error finding container 641c49944e551bf71c97262a300dff3433d0215f4775b55c01ecda24d49d9156: Status 404 returned error can't find the container with id 641c49944e551bf71c97262a300dff3433d0215f4775b55c01ecda24d49d9156
Jul 30 00:17:58.140264 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:58.140205    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1332083_d23f_44ac_a0f2_8d6046a0fe48.slice/crio-1f011858e27675f1356caaf43c76d0f2abe9e4115c529d725ec9c4c70b6b44b9 WatchSource:0}: Error finding container 1f011858e27675f1356caaf43c76d0f2abe9e4115c529d725ec9c4c70b6b44b9: Status 404 returned error can't find the container with id 1f011858e27675f1356caaf43c76d0f2abe9e4115c529d725ec9c4c70b6b44b9
Jul 30 00:17:58.141258 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:17:58.141234    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182ce8f5_d95e_479e_9adf_a5571616b663.slice/crio-849602f4c18ff08816ee552b7521b6770b09a4646e68956dce1b1cb3cd7a6a50 WatchSource:0}: Error finding container 849602f4c18ff08816ee552b7521b6770b09a4646e68956dce1b1cb3cd7a6a50: Status 404 returned error can't find the container with id 849602f4c18ff08816ee552b7521b6770b09a4646e68956dce1b1cb3cd7a6a50
Jul 30 00:17:58.153121 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:58.153069    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 00:12:57 +0000 UTC" deadline="2028-02-09 22:34:37.948081501 +0000 UTC"
Jul 30 00:17:58.153121 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:58.153121    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13438h16m39.794963626s"
Jul 30 00:17:58.316898 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:58.316862    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:17:58.317049 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:58.316995    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:17:58.317049 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:58.317027    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:17:58.317049 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:58.317016    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hg9rq" podUID="3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0"
Jul 30 00:17:58.317203 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:58.317110    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6ns2" podUID="a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1"
Jul 30 00:17:58.317203 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:58.317188    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86f4q" podUID="ebd8b209-a805-4976-bb8c-24cba1f01d86"
Jul 30 00:17:58.327494 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:58.327456    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" event={"ID":"260636c800f3b5abe8a8cc948b2e682e","Type":"ContainerStarted","Data":"cc1cf8d651b97300f687ebf8c52725878dedfdf29127c2bce71d84747c6ce2c9"}
Jul 30 00:17:58.328788 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:58.328759    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cwm68" event={"ID":"c1856c95-b8d8-4303-9ec9-04f3f8f3dec2","Type":"ContainerStarted","Data":"b61cd07a2975ef71dec6d45830b5528e324451b38581da2e488043e57294e913"}
Jul 30 00:17:58.330135 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:58.330102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-slhpw" event={"ID":"fc610fab-9658-40eb-8527-081a0008d70f","Type":"ContainerStarted","Data":"70fd9eeff70c5b2d809e7aa7d6debf830181bec4c2da889be491f267df30d0a7"}
Jul 30 00:17:58.331494 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:58.331458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9bqqn" event={"ID":"7471b635-17bb-4f3a-b41f-ee9fa0f9871e","Type":"ContainerStarted","Data":"6c0f75347d497b35787a8b568190db217ff66566ee45fd9c2c3fbd596fcc6b11"}
Jul 30 00:17:58.332763 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:58.332737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2jclk" event={"ID":"318afb78-6a56-4524-8c63-8a3c2b223271","Type":"ContainerStarted","Data":"d226c82559c574d2ee6deb8382983ce5d0450dc9eeb4a4ed67778be9526b5666"}
Jul 30 00:17:58.334086 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:58.334055    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qbkcd" event={"ID":"182ce8f5-d95e-479e-9adf-a5571616b663","Type":"ContainerStarted","Data":"849602f4c18ff08816ee552b7521b6770b09a4646e68956dce1b1cb3cd7a6a50"}
Jul 30 00:17:58.335480 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:58.335453    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rglkm" event={"ID":"c1332083-d23f-44ac-a0f2-8d6046a0fe48","Type":"ContainerStarted","Data":"1f011858e27675f1356caaf43c76d0f2abe9e4115c529d725ec9c4c70b6b44b9"}
Jul 30 00:17:58.336848 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:58.336818    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pwx6g" event={"ID":"836811a6-7584-4818-884a-59ce980afe22","Type":"ContainerStarted","Data":"44e34db8f4c53b8bc786420d46c5195afcdef656d14c1f61f7405e7f103d5e5b"}
Jul 30 00:17:58.338052 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:58.338028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfkf9" event={"ID":"7f6d919d-802a-4eb8-a7ec-948626dfc26b","Type":"ContainerStarted","Data":"641c49944e551bf71c97262a300dff3433d0215f4775b55c01ecda24d49d9156"}
Jul 30 00:17:58.339348 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:58.339323    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm" event={"ID":"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0","Type":"ContainerStarted","Data":"eec8c4cf48997e0889e57313d55dafc4623b15ba7a5602acb500baa66efea53d"}
Jul 30 00:17:58.342792 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:58.342740    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" podStartSLOduration=1.342721715 podStartE2EDuration="1.342721715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:17:58.342681256 +0000 UTC m=+3.639948192" watchObservedRunningTime="2026-07-30 00:17:58.342721715 +0000 UTC m=+3.639988655"
Jul 30 00:17:58.742957 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:58.742652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret\") pod \"global-pull-secret-syncer-86f4q\" (UID: \"ebd8b209-a805-4976-bb8c-24cba1f01d86\") " pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:17:58.743156 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:58.742972    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 00:17:58.743156 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:58.743017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs\") pod \"network-metrics-daemon-b6ns2\" (UID: \"a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1\") " pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:17:58.743156 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:58.743057    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret podName:ebd8b209-a805-4976-bb8c-24cba1f01d86 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:00.74303358 +0000 UTC m=+6.040300512 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret") pod "global-pull-secret-syncer-86f4q" (UID: "ebd8b209-a805-4976-bb8c-24cba1f01d86") : object "kube-system"/"original-pull-secret" not registered
Jul 30 00:17:58.743156 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:58.743121    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 00:17:58.743369 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:58.743183    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs podName:a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:00.743164414 +0000 UTC m=+6.040431345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs") pod "network-metrics-daemon-b6ns2" (UID: "a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 00:17:58.947090 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:58.946210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmm96\" (UniqueName: \"kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96\") pod \"network-check-target-hg9rq\" (UID: \"3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0\") " pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:17:58.947090 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:58.946412    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 00:17:58.947090 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:58.946430    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 00:17:58.947090 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:58.946441    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bmm96 for pod openshift-network-diagnostics/network-check-target-hg9rq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 00:17:58.947090 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:17:58.946493    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96 podName:3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:00.94647739 +0000 UTC m=+6.243744316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bmm96" (UniqueName: "kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96") pod "network-check-target-hg9rq" (UID: "3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 00:17:59.153321 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:59.153241    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 00:12:57 +0000 UTC" deadline="2028-01-10 18:40:36.400639623 +0000 UTC"
Jul 30 00:17:59.153321 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:59.153303    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12714h22m37.24734126s"
Jul 30 00:17:59.351961 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:17:59.351877    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" event={"ID":"45dcbd02e4d8cba07fa3d3a342359b2c","Type":"ContainerStarted","Data":"16359163b57bdb99e6735a08b283ecfb35578d51defc8689050f798276301672"}
Jul 30 00:18:00.316585 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:00.316547    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:00.316585 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:00.316587    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:00.317171 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:00.316714    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6ns2" podUID="a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1"
Jul 30 00:18:00.317171 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:00.316877    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86f4q" podUID="ebd8b209-a805-4976-bb8c-24cba1f01d86"
Jul 30 00:18:00.317171 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:00.316547    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:00.317171 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:00.316982    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hg9rq" podUID="3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0"
Jul 30 00:18:00.381359 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:00.380678    2579 generic.go:356] "Generic (PLEG): container finished" podID="45dcbd02e4d8cba07fa3d3a342359b2c" containerID="16359163b57bdb99e6735a08b283ecfb35578d51defc8689050f798276301672" exitCode=0
Jul 30 00:18:00.381359 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:00.380734    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" event={"ID":"45dcbd02e4d8cba07fa3d3a342359b2c","Type":"ContainerDied","Data":"16359163b57bdb99e6735a08b283ecfb35578d51defc8689050f798276301672"}
Jul 30 00:18:00.763199 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:00.762983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs\") pod \"network-metrics-daemon-b6ns2\" (UID: \"a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1\") " pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:00.763199 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:00.763076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret\") pod \"global-pull-secret-syncer-86f4q\" (UID: \"ebd8b209-a805-4976-bb8c-24cba1f01d86\") " pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:00.763492 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:00.763223    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 00:18:00.763492 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:00.763313    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret podName:ebd8b209-a805-4976-bb8c-24cba1f01d86 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:04.763273596 +0000 UTC m=+10.060540510 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret") pod "global-pull-secret-syncer-86f4q" (UID: "ebd8b209-a805-4976-bb8c-24cba1f01d86") : object "kube-system"/"original-pull-secret" not registered
Jul 30 00:18:00.763996 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:00.763892    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 00:18:00.763996 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:00.763961    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs podName:a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:04.763940693 +0000 UTC m=+10.061207608 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs") pod "network-metrics-daemon-b6ns2" (UID: "a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 00:18:00.964983 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:00.964167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmm96\" (UniqueName: \"kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96\") pod \"network-check-target-hg9rq\" (UID: \"3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0\") " pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:00.964983 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:00.964404    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 00:18:00.964983 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:00.964430    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 00:18:00.964983 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:00.964444    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bmm96 for pod openshift-network-diagnostics/network-check-target-hg9rq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 00:18:00.964983 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:00.964511    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96 podName:3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:04.964489468 +0000 UTC m=+10.261756394 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bmm96" (UniqueName: "kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96") pod "network-check-target-hg9rq" (UID: "3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 00:18:01.389617 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:01.389521    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" event={"ID":"45dcbd02e4d8cba07fa3d3a342359b2c","Type":"ContainerStarted","Data":"15192f3c62cd7f5d6e3a835f0f904d324d5083928113cc40a493da24cadf2557"}
Jul 30 00:18:01.410187 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:01.409672    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" podStartSLOduration=4.409648083 podStartE2EDuration="4.409648083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:01.408493122 +0000 UTC m=+6.705760059" watchObservedRunningTime="2026-07-30 00:18:01.409648083 +0000 UTC m=+6.706915020"
Jul 30 00:18:02.318078 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:02.317099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:02.318078 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:02.317242    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86f4q" podUID="ebd8b209-a805-4976-bb8c-24cba1f01d86"
Jul 30 00:18:02.318078 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:02.317708    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:02.318078 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:02.317840    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6ns2" podUID="a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1"
Jul 30 00:18:02.318078 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:02.317905    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:02.318078 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:02.317985    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hg9rq" podUID="3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0"
Jul 30 00:18:04.316756 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:04.316719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:04.317285 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:04.316854    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hg9rq" podUID="3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0"
Jul 30 00:18:04.317285 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:04.317191    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:04.317422 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:04.317288    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86f4q" podUID="ebd8b209-a805-4976-bb8c-24cba1f01d86"
Jul 30 00:18:04.317422 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:04.317329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:04.317422 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:04.317385    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6ns2" podUID="a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1"
Jul 30 00:18:04.797498 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:04.797154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs\") pod \"network-metrics-daemon-b6ns2\" (UID: \"a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1\") " pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:04.797498 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:04.797264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret\") pod \"global-pull-secret-syncer-86f4q\" (UID: \"ebd8b209-a805-4976-bb8c-24cba1f01d86\") " pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:04.797498 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:04.797356    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 00:18:04.797498 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:04.797378    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 00:18:04.797498 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:04.797438    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs podName:a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:12.797416018 +0000 UTC m=+18.094682943 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs") pod "network-metrics-daemon-b6ns2" (UID: "a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 00:18:04.797498 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:04.797458    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret podName:ebd8b209-a805-4976-bb8c-24cba1f01d86 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:12.797447926 +0000 UTC m=+18.094714841 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret") pod "global-pull-secret-syncer-86f4q" (UID: "ebd8b209-a805-4976-bb8c-24cba1f01d86") : object "kube-system"/"original-pull-secret" not registered
Jul 30 00:18:04.999285 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:04.999230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmm96\" (UniqueName: \"kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96\") pod \"network-check-target-hg9rq\" (UID: \"3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0\") " pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:04.999508 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:04.999472    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 00:18:04.999508 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:04.999497    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 00:18:04.999631 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:04.999514    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bmm96 for pod openshift-network-diagnostics/network-check-target-hg9rq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 00:18:04.999631 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:04.999585    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96 podName:3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:12.999560034 +0000 UTC m=+18.296826968 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bmm96" (UniqueName: "kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96") pod "network-check-target-hg9rq" (UID: "3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 00:18:06.316769 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:06.316730    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:06.317314 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:06.316875    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hg9rq" podUID="3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0"
Jul 30 00:18:06.317314 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:06.317161    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:06.317314 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:06.317171    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:06.317479 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:06.317320    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6ns2" podUID="a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1"
Jul 30 00:18:06.317479 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:06.317413    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86f4q" podUID="ebd8b209-a805-4976-bb8c-24cba1f01d86"
Jul 30 00:18:08.316364 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:08.316333    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:08.316866 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:08.316459    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:08.316866 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:08.316471    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86f4q" podUID="ebd8b209-a805-4976-bb8c-24cba1f01d86"
Jul 30 00:18:08.316866 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:08.316497    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:08.316866 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:08.316586    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hg9rq" podUID="3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0"
Jul 30 00:18:08.316866 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:08.316672    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6ns2" podUID="a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1"
Jul 30 00:18:10.316425 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:10.316392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:10.316888 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:10.316508    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:10.316888 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:10.316516    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86f4q" podUID="ebd8b209-a805-4976-bb8c-24cba1f01d86"
Jul 30 00:18:10.316888 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:10.316545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:10.316888 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:10.316643    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6ns2" podUID="a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1"
Jul 30 00:18:10.316888 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:10.316723    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hg9rq" podUID="3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0"
Jul 30 00:18:12.316102 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:12.316063    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:12.316102 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:12.316099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:12.316630 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:12.316182    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:12.316630 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:12.316202    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hg9rq" podUID="3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0"
Jul 30 00:18:12.316630 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:12.316311    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6ns2" podUID="a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1"
Jul 30 00:18:12.316630 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:12.316399    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86f4q" podUID="ebd8b209-a805-4976-bb8c-24cba1f01d86"
Jul 30 00:18:12.863101 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:12.863070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret\") pod \"global-pull-secret-syncer-86f4q\" (UID: \"ebd8b209-a805-4976-bb8c-24cba1f01d86\") " pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:12.863253 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:12.863123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs\") pod \"network-metrics-daemon-b6ns2\" (UID: \"a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1\") " pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:12.863253 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:12.863220    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 00:18:12.863253 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:12.863221    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 00:18:12.863364 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:12.863295    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs podName:a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:28.863272619 +0000 UTC m=+34.160539549 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs") pod "network-metrics-daemon-b6ns2" (UID: "a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 00:18:12.863364 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:12.863310    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret podName:ebd8b209-a805-4976-bb8c-24cba1f01d86 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:28.86330346 +0000 UTC m=+34.160570369 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret") pod "global-pull-secret-syncer-86f4q" (UID: "ebd8b209-a805-4976-bb8c-24cba1f01d86") : object "kube-system"/"original-pull-secret" not registered
Jul 30 00:18:13.064635 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:13.064549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmm96\" (UniqueName: \"kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96\") pod \"network-check-target-hg9rq\" (UID: \"3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0\") " pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:13.064791 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:13.064738    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 00:18:13.064791 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:13.064764    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 00:18:13.064791 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:13.064775    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bmm96 for pod openshift-network-diagnostics/network-check-target-hg9rq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 00:18:13.064923 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:13.064840    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96 podName:3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:29.064819482 +0000 UTC m=+34.362086394 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bmm96" (UniqueName: "kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96") pod "network-check-target-hg9rq" (UID: "3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 00:18:14.316461 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:14.316424    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:14.316461 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:14.316463    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:14.316997 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:14.316463    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:14.316997 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:14.316564    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6ns2" podUID="a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1"
Jul 30 00:18:14.317097 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:14.317071    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86f4q" podUID="ebd8b209-a805-4976-bb8c-24cba1f01d86"
Jul 30 00:18:14.317229 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:14.317201    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hg9rq" podUID="3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0"
Jul 30 00:18:16.317106 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:16.317068    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:16.317625 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:16.317086    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:16.317625 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:16.317207    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hg9rq" podUID="3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0"
Jul 30 00:18:16.317625 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:16.317258    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:16.317625 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:16.317351    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6ns2" podUID="a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1"
Jul 30 00:18:16.317625 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:16.317432    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86f4q" podUID="ebd8b209-a805-4976-bb8c-24cba1f01d86"
Jul 30 00:18:17.425152 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:17.424788    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:18:17.425838 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:17.425519    2579 generic.go:356] "Generic (PLEG): container finished" podID="fc610fab-9658-40eb-8527-081a0008d70f" containerID="d93299ddc2022507c84d4dc90438dca4909e5b487e2db2ff54268c9cb391df91" exitCode=1
Jul 30 00:18:17.425838 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:17.425588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-slhpw" event={"ID":"fc610fab-9658-40eb-8527-081a0008d70f","Type":"ContainerStarted","Data":"fc7678868c72122ff0d07f43b09ba098a34bfa24c6f4fd1ec30aff5225c976f1"}
Jul 30 00:18:17.425838 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:17.425655    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-slhpw" event={"ID":"fc610fab-9658-40eb-8527-081a0008d70f","Type":"ContainerDied","Data":"d93299ddc2022507c84d4dc90438dca4909e5b487e2db2ff54268c9cb391df91"}
Jul 30 00:18:17.425838 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:17.425694    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-slhpw" event={"ID":"fc610fab-9658-40eb-8527-081a0008d70f","Type":"ContainerStarted","Data":"a9676bca68db9f9241bfbbe6388fa35365aad31a05c15b8af54c76f3c3967f2e"}
Jul 30 00:18:17.426810 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:17.426787    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9bqqn" event={"ID":"7471b635-17bb-4f3a-b41f-ee9fa0f9871e","Type":"ContainerStarted","Data":"47d1478b3f6eaad194039e87d7832a0e41a9161e5247dd41668f4e8a98545849"}
Jul 30 00:18:17.428186 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:17.428165    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2jclk" event={"ID":"318afb78-6a56-4524-8c63-8a3c2b223271","Type":"ContainerStarted","Data":"e58f0200e8ed2c2e74e2a6d4d1ef51aa168c1d3b32c5a0d4214af45f08b549c6"}
Jul 30 00:18:17.429437 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:17.429415    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qbkcd" event={"ID":"182ce8f5-d95e-479e-9adf-a5571616b663","Type":"ContainerStarted","Data":"a546dccd0eb498fbfc69db349c1cf62663c8a4d3f9ab5f0c89e8810b74afab55"}
Jul 30 00:18:17.430958 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:17.430935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rglkm" event={"ID":"c1332083-d23f-44ac-a0f2-8d6046a0fe48","Type":"ContainerStarted","Data":"87dd560a114328ba5bd653e24e93735716b8f4db38df86d741cde781a45c4ff3"}
Jul 30 00:18:17.432397 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:17.432377    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pwx6g" event={"ID":"836811a6-7584-4818-884a-59ce980afe22","Type":"ContainerStarted","Data":"2fe032421430a0d0bfe8aed7523557bae9fbd41d646c46ab9e26f1a2c5e067cf"}
Jul 30 00:18:17.434036 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:17.434010    2579 generic.go:356] "Generic (PLEG): container finished" podID="7f6d919d-802a-4eb8-a7ec-948626dfc26b" containerID="78db93cb307bfa3706b36e5e7a41f6ce1b8a125bf6889ee533bbd56cf69a0654" exitCode=0
Jul 30 00:18:17.434131 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:17.434102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfkf9" event={"ID":"7f6d919d-802a-4eb8-a7ec-948626dfc26b","Type":"ContainerDied","Data":"78db93cb307bfa3706b36e5e7a41f6ce1b8a125bf6889ee533bbd56cf69a0654"}
Jul 30 00:18:17.435671 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:17.435651    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm" event={"ID":"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0","Type":"ContainerStarted","Data":"8ed628c4ad63d1d2a71ac0f73c12a61e750d861b22054b1d0ba6078ce698dacb"}
Jul 30 00:18:17.443563 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:17.443525    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9bqqn" podStartSLOduration=3.5800780420000002 podStartE2EDuration="22.443494599s" totalImagesPullingTime="18.863416557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:17.443468734 +0000 UTC m=+22.740735667" watchObservedRunningTime="2026-07-30 00:18:17.443494599 +0000 UTC m=+22.740761514"
Jul 30 00:18:17.479480 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:17.479441    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2jclk" podStartSLOduration=3.667574039 podStartE2EDuration="22.479428307s" totalImagesPullingTime="18.811854268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:17.479401092 +0000 UTC m=+22.776668023" watchObservedRunningTime="2026-07-30 00:18:17.479428307 +0000 UTC m=+22.776695238"
Jul 30 00:18:17.493282 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:17.493205    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qbkcd" podStartSLOduration=7.872366564 podStartE2EDuration="22.493186843s" totalImagesPullingTime="14.620820279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:17.492825647 +0000 UTC m=+22.790092579" watchObservedRunningTime="2026-07-30 00:18:17.493186843 +0000 UTC m=+22.790453776"
Jul 30 00:18:17.508615 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:17.508572    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rglkm" podStartSLOduration=3.694692493 podStartE2EDuration="22.50855917s" totalImagesPullingTime="18.813866677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:17.507945802 +0000 UTC m=+22.805212727" watchObservedRunningTime="2026-07-30 00:18:17.50855917 +0000 UTC m=+22.805826102"
Jul 30 00:18:17.522873 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:17.522838    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pwx6g" podStartSLOduration=3.695285584 podStartE2EDuration="22.522825771s" totalImagesPullingTime="18.827540187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:17.522707853 +0000 UTC m=+22.819974786" watchObservedRunningTime="2026-07-30 00:18:17.522825771 +0000 UTC m=+22.820092703"
Jul 30 00:18:18.316425 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:18.316392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:18.316641 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:18.316392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:18.316641 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:18.316517    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86f4q" podUID="ebd8b209-a805-4976-bb8c-24cba1f01d86"
Jul 30 00:18:18.316641 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:18.316401    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:18.316641 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:18.316589    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hg9rq" podUID="3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0"
Jul 30 00:18:18.316851 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:18.316664    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6ns2" podUID="a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1"
Jul 30 00:18:18.439380 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:18.439338    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cwm68" event={"ID":"c1856c95-b8d8-4303-9ec9-04f3f8f3dec2","Type":"ContainerStarted","Data":"9918a973b1bd7aa911a82ee354395ca821ecc501e25dc708182f9641dd138d35"}
Jul 30 00:18:18.442819 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:18.442798    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:18:18.443561 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:18.443526    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-slhpw" event={"ID":"fc610fab-9658-40eb-8527-081a0008d70f","Type":"ContainerStarted","Data":"6594ebea1e31d007b7a1db7c5a2278502a8e4fbe1629cc18d9aaf1b449d44bb1"}
Jul 30 00:18:18.443706 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:18.443608    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-slhpw" event={"ID":"fc610fab-9658-40eb-8527-081a0008d70f","Type":"ContainerStarted","Data":"86a94bfba83a11c984483df5a317caad6632ae816277c7e888c0ababf298b7d2"}
Jul 30 00:18:18.443706 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:18.443666    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-slhpw" event={"ID":"fc610fab-9658-40eb-8527-081a0008d70f","Type":"ContainerStarted","Data":"063a47f1bf2f347be585b762d9687cd927526a94b4bd8597b7aa3d50438fefaf"}
Jul 30 00:18:18.454146 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:18.454094    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cwm68" podStartSLOduration=4.645716014 podStartE2EDuration="23.454080764s" totalImagesPullingTime="18.80836475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:18.453688559 +0000 UTC m=+23.750955491" watchObservedRunningTime="2026-07-30 00:18:18.454080764 +0000 UTC m=+23.751347756"
Jul 30 00:18:18.601478 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:18.601450    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 30 00:18:19.190529 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:19.190419    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T00:18:18.601474392Z","UUID":"9b260cc9-7c42-4b71-9f76-9667853788e2","Handler":null,"Name":"","Endpoint":""}
Jul 30 00:18:19.191982 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:19.191956    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 30 00:18:19.192105 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:19.191994    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 30 00:18:19.448845 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:19.448751    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm" event={"ID":"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0","Type":"ContainerStarted","Data":"cb71992a11c8ee9cf05b889aea34cf80d400fdbf2146f55d316e666527688cff"}
Jul 30 00:18:20.316148 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:20.316112    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:20.316372 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:20.316112    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:20.316372 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:20.316239    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hg9rq" podUID="3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0"
Jul 30 00:18:20.316372 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:20.316112    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:20.316372 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:20.316332    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86f4q" podUID="ebd8b209-a805-4976-bb8c-24cba1f01d86"
Jul 30 00:18:20.316567 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:20.316401    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6ns2" podUID="a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1"
Jul 30 00:18:20.454790 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:20.454762    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:18:20.455283 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:20.455182    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-slhpw" event={"ID":"fc610fab-9658-40eb-8527-081a0008d70f","Type":"ContainerStarted","Data":"eac9d01c3c4e82cf1fcdb2bf546fe599b8799e90cf7de8d47289b57e79fefb4b"}
Jul 30 00:18:20.457705 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:20.457671    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm" event={"ID":"8a72c231-04bd-4d8c-a0dd-c59d5a61c4a0","Type":"ContainerStarted","Data":"2c69e0ff34451e0d4f2d10b3313687fb5472879b2a5d3471d09c6a5d116c05a7"}
Jul 30 00:18:20.477230 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:20.477175    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgfxm" podStartSLOduration=4.392694098 podStartE2EDuration="25.477159652s" totalImagesPullingTime="21.084465554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:20.476419296 +0000 UTC m=+25.773686229" watchObservedRunningTime="2026-07-30 00:18:20.477159652 +0000 UTC m=+25.774426584"
Jul 30 00:18:21.132926 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:21.132895    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qbkcd"
Jul 30 00:18:21.133890 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:21.133658    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qbkcd"
Jul 30 00:18:21.461090 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:21.461002    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qbkcd"
Jul 30 00:18:21.461582 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:21.461558    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qbkcd"
Jul 30 00:18:22.316386 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:22.316355    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:22.316548 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:22.316358    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:22.316548 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:22.316458    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86f4q" podUID="ebd8b209-a805-4976-bb8c-24cba1f01d86"
Jul 30 00:18:22.316640 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:22.316546    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6ns2" podUID="a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1"
Jul 30 00:18:22.316640 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:22.316358    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:22.316712 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:22.316621    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hg9rq" podUID="3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0"
Jul 30 00:18:22.465591 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:22.465565    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:18:22.466443 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:22.465929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-slhpw" event={"ID":"fc610fab-9658-40eb-8527-081a0008d70f","Type":"ContainerStarted","Data":"c8513c90c4bb4e7c65ac9d5cf5474c4206325bc9922817cc2b53a4370fad101f"}
Jul 30 00:18:22.466443 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:22.466185    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:18:22.466443 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:22.466433    2579 scope.go:117] "RemoveContainer" containerID="d93299ddc2022507c84d4dc90438dca4909e5b487e2db2ff54268c9cb391df91"
Jul 30 00:18:22.467985 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:22.467958    2579 generic.go:356] "Generic (PLEG): container finished" podID="7f6d919d-802a-4eb8-a7ec-948626dfc26b" containerID="e19460af135829a9074971b4a3887eb460790b26d79c5964f85c0c86093ef5f5" exitCode=0
Jul 30 00:18:22.468087 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:22.468045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfkf9" event={"ID":"7f6d919d-802a-4eb8-a7ec-948626dfc26b","Type":"ContainerDied","Data":"e19460af135829a9074971b4a3887eb460790b26d79c5964f85c0c86093ef5f5"}
Jul 30 00:18:22.485710 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:22.485687    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:18:23.472442 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:23.472158    2579 generic.go:356] "Generic (PLEG): container finished" podID="7f6d919d-802a-4eb8-a7ec-948626dfc26b" containerID="9a1321ba2dc4c1bef84a4ca3620bd1355ee942730966eb2fc0afe79d98839e4b" exitCode=0
Jul 30 00:18:23.472442 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:23.472264    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfkf9" event={"ID":"7f6d919d-802a-4eb8-a7ec-948626dfc26b","Type":"ContainerDied","Data":"9a1321ba2dc4c1bef84a4ca3620bd1355ee942730966eb2fc0afe79d98839e4b"}
Jul 30 00:18:23.476297 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:23.476280    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:18:23.476623 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:23.476597    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-slhpw" event={"ID":"fc610fab-9658-40eb-8527-081a0008d70f","Type":"ContainerStarted","Data":"b693f608d10f0861095372e0a3d036bc555fa94f309fffed07906e5440a54791"}
Jul 30 00:18:23.476914 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:23.476896    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:18:23.476989 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:23.476954    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:18:23.492187 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:23.492167    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:18:23.519990 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:23.519945    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-slhpw" podStartSLOduration=9.663023121 podStartE2EDuration="28.519931413s" totalImagesPullingTime="18.856908292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:23.519343566 +0000 UTC m=+28.816610497" watchObservedRunningTime="2026-07-30 00:18:23.519931413 +0000 UTC m=+28.817198345"
Jul 30 00:18:24.316137 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:24.316109    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:24.316137 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:24.316130    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:24.316288 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:24.316225    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86f4q" podUID="ebd8b209-a805-4976-bb8c-24cba1f01d86"
Jul 30 00:18:24.316321 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:24.316280    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:24.316453 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:24.316426    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6ns2" podUID="a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1"
Jul 30 00:18:24.316501 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:24.316487    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hg9rq" podUID="3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0"
Jul 30 00:18:24.481001 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:24.480972    2579 generic.go:356] "Generic (PLEG): container finished" podID="7f6d919d-802a-4eb8-a7ec-948626dfc26b" containerID="8efd88eeb7e141f69f2aa484b27af1c43f8ff6cb35e90eca14778e25b6dd97da" exitCode=0
Jul 30 00:18:24.481436 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:24.481048    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfkf9" event={"ID":"7f6d919d-802a-4eb8-a7ec-948626dfc26b","Type":"ContainerDied","Data":"8efd88eeb7e141f69f2aa484b27af1c43f8ff6cb35e90eca14778e25b6dd97da"}
Jul 30 00:18:26.046338 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:26.046304    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-86f4q"]
Jul 30 00:18:26.046783 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:26.046489    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:26.046783 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:26.046608    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86f4q" podUID="ebd8b209-a805-4976-bb8c-24cba1f01d86"
Jul 30 00:18:26.050557 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:26.050286    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hg9rq"]
Jul 30 00:18:26.050557 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:26.050404    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:26.050557 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:26.050510    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hg9rq" podUID="3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0"
Jul 30 00:18:26.051222 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:26.051189    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b6ns2"]
Jul 30 00:18:26.051347 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:26.051319    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:26.051447 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:26.051424    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6ns2" podUID="a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1"
Jul 30 00:18:27.316798 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:27.316764    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:27.317294 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:27.316913    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86f4q" podUID="ebd8b209-a805-4976-bb8c-24cba1f01d86"
Jul 30 00:18:28.316777 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:28.316499    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:28.317016 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:28.316559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:28.317016 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:28.316845    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6ns2" podUID="a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1"
Jul 30 00:18:28.317016 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:28.316961    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hg9rq" podUID="3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0"
Jul 30 00:18:28.892412 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:28.892386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs\") pod \"network-metrics-daemon-b6ns2\" (UID: \"a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1\") " pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:28.892487 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:28.892460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret\") pod \"global-pull-secret-syncer-86f4q\" (UID: \"ebd8b209-a805-4976-bb8c-24cba1f01d86\") " pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:28.892542 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:28.892524    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 00:18:28.892594 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:28.892581    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 00:18:28.892634 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:28.892585    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs podName:a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1 nodeName:}" failed. No retries permitted until 2026-07-30 00:19:00.892569694 +0000 UTC m=+66.189836604 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs") pod "network-metrics-daemon-b6ns2" (UID: "a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 00:18:28.892676 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:28.892655    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret podName:ebd8b209-a805-4976-bb8c-24cba1f01d86 nodeName:}" failed. No retries permitted until 2026-07-30 00:19:00.89263883 +0000 UTC m=+66.189905740 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret") pod "global-pull-secret-syncer-86f4q" (UID: "ebd8b209-a805-4976-bb8c-24cba1f01d86") : object "kube-system"/"original-pull-secret" not registered
Jul 30 00:18:29.093581 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:29.093556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmm96\" (UniqueName: \"kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96\") pod \"network-check-target-hg9rq\" (UID: \"3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0\") " pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:29.093739 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:29.093721    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 00:18:29.093778 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:29.093743    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 00:18:29.093778 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:29.093753    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bmm96 for pod openshift-network-diagnostics/network-check-target-hg9rq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 00:18:29.093843 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:29.093805    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96 podName:3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0 nodeName:}" failed. No retries permitted until 2026-07-30 00:19:01.09378973 +0000 UTC m=+66.391056642 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bmm96" (UniqueName: "kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96") pod "network-check-target-hg9rq" (UID: "3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 00:18:29.316964 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:29.316895    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:29.317129 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:29.317008    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86f4q" podUID="ebd8b209-a805-4976-bb8c-24cba1f01d86"
Jul 30 00:18:29.496007 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:29.495964    2579 generic.go:356] "Generic (PLEG): container finished" podID="7f6d919d-802a-4eb8-a7ec-948626dfc26b" containerID="22ae7096764d32e94190c32caaa29ca103ed3529c311a4aaeb1cd522a0fd8fba" exitCode=0
Jul 30 00:18:29.496225 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:29.496015    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfkf9" event={"ID":"7f6d919d-802a-4eb8-a7ec-948626dfc26b","Type":"ContainerDied","Data":"22ae7096764d32e94190c32caaa29ca103ed3529c311a4aaeb1cd522a0fd8fba"}
Jul 30 00:18:29.949203 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:29.949131    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeReady"
Jul 30 00:18:29.949445 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:29.949266    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 30 00:18:30.005481 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.005453    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7pwht"]
Jul 30 00:18:30.032745 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.032717    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9v8g2"]
Jul 30 00:18:30.032902 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.032825    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7pwht"
Jul 30 00:18:30.035444 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.035422    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 30 00:18:30.035568 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.035483    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 30 00:18:30.035748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.035730    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 30 00:18:30.035813 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.035750    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qvszd\""
Jul 30 00:18:30.048769 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.048745    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7pwht"]
Jul 30 00:18:30.048769 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.048774    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9v8g2"]
Jul 30 00:18:30.048916 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.048799    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lcr97"]
Jul 30 00:18:30.048968 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.048948    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9v8g2"
Jul 30 00:18:30.051618 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.051599    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 30 00:18:30.051735 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.051656    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 30 00:18:30.051735 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.051696    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 30 00:18:30.051998 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.051984    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mbp9q\""
Jul 30 00:18:30.052060 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.052046    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 30 00:18:30.063335 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.063316    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lcr97"]
Jul 30 00:18:30.063428 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.063400    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lcr97"
Jul 30 00:18:30.066430 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.066408    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 30 00:18:30.066543 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.066489    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 30 00:18:30.066543 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.066506    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-twxtz\""
Jul 30 00:18:30.201879 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.201816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39fd448-9f15-4275-86cf-5d233023c5a5-metrics-tls\") pod \"dns-default-lcr97\" (UID: \"b39fd448-9f15-4275-86cf-5d233023c5a5\") " pod="openshift-dns/dns-default-lcr97"
Jul 30 00:18:30.201879 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.201855    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br7r5\" (UniqueName: \"kubernetes.io/projected/b39fd448-9f15-4275-86cf-5d233023c5a5-kube-api-access-br7r5\") pod \"dns-default-lcr97\" (UID: \"b39fd448-9f15-4275-86cf-5d233023c5a5\") " pod="openshift-dns/dns-default-lcr97"
Jul 30 00:18:30.201879 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.201874    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b39fd448-9f15-4275-86cf-5d233023c5a5-config-volume\") pod \"dns-default-lcr97\" (UID: \"b39fd448-9f15-4275-86cf-5d233023c5a5\") " pod="openshift-dns/dns-default-lcr97"
Jul 30 00:18:30.202096 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.201898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79gwx\" (UniqueName: \"kubernetes.io/projected/466f60c6-0f78-4017-a90f-87489f71021c-kube-api-access-79gwx\") pod \"insights-runtime-extractor-9v8g2\" (UID: \"466f60c6-0f78-4017-a90f-87489f71021c\") " pod="openshift-insights/insights-runtime-extractor-9v8g2"
Jul 30 00:18:30.202096 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.201950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/466f60c6-0f78-4017-a90f-87489f71021c-data-volume\") pod \"insights-runtime-extractor-9v8g2\" (UID: \"466f60c6-0f78-4017-a90f-87489f71021c\") " pod="openshift-insights/insights-runtime-extractor-9v8g2"
Jul 30 00:18:30.202096 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.201965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/466f60c6-0f78-4017-a90f-87489f71021c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9v8g2\" (UID: \"466f60c6-0f78-4017-a90f-87489f71021c\") " pod="openshift-insights/insights-runtime-extractor-9v8g2"
Jul 30 00:18:30.202096 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.201983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/466f60c6-0f78-4017-a90f-87489f71021c-crio-socket\") pod \"insights-runtime-extractor-9v8g2\" (UID: \"466f60c6-0f78-4017-a90f-87489f71021c\") " pod="openshift-insights/insights-runtime-extractor-9v8g2"
Jul 30 00:18:30.202096 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.202005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b39fd448-9f15-4275-86cf-5d233023c5a5-tmp-dir\") pod \"dns-default-lcr97\" (UID: \"b39fd448-9f15-4275-86cf-5d233023c5a5\") " pod="openshift-dns/dns-default-lcr97"
Jul 30 00:18:30.202096 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.202025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c0002be-c29e-4574-9d66-739390a6a6d3-cert\") pod \"ingress-canary-7pwht\" (UID: \"2c0002be-c29e-4574-9d66-739390a6a6d3\") " pod="openshift-ingress-canary/ingress-canary-7pwht"
Jul 30 00:18:30.202096 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.202040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzkzr\" (UniqueName: \"kubernetes.io/projected/2c0002be-c29e-4574-9d66-739390a6a6d3-kube-api-access-nzkzr\") pod \"ingress-canary-7pwht\" (UID: \"2c0002be-c29e-4574-9d66-739390a6a6d3\") " pod="openshift-ingress-canary/ingress-canary-7pwht"
Jul 30 00:18:30.202325 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.202112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/466f60c6-0f78-4017-a90f-87489f71021c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9v8g2\" (UID: \"466f60c6-0f78-4017-a90f-87489f71021c\") " pod="openshift-insights/insights-runtime-extractor-9v8g2"
Jul 30 00:18:30.303132 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.303107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/466f60c6-0f78-4017-a90f-87489f71021c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9v8g2\" (UID: \"466f60c6-0f78-4017-a90f-87489f71021c\") " pod="openshift-insights/insights-runtime-extractor-9v8g2"
Jul 30 00:18:30.303265 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.303147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39fd448-9f15-4275-86cf-5d233023c5a5-metrics-tls\") pod \"dns-default-lcr97\" (UID: \"b39fd448-9f15-4275-86cf-5d233023c5a5\") " pod="openshift-dns/dns-default-lcr97"
Jul 30 00:18:30.303265 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.303170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br7r5\" (UniqueName: \"kubernetes.io/projected/b39fd448-9f15-4275-86cf-5d233023c5a5-kube-api-access-br7r5\") pod \"dns-default-lcr97\" (UID: \"b39fd448-9f15-4275-86cf-5d233023c5a5\") " pod="openshift-dns/dns-default-lcr97"
Jul 30 00:18:30.303265 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.303191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b39fd448-9f15-4275-86cf-5d233023c5a5-config-volume\") pod \"dns-default-lcr97\" (UID: \"b39fd448-9f15-4275-86cf-5d233023c5a5\") " pod="openshift-dns/dns-default-lcr97"
Jul 30 00:18:30.303265 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.303218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79gwx\" (UniqueName: \"kubernetes.io/projected/466f60c6-0f78-4017-a90f-87489f71021c-kube-api-access-79gwx\") pod \"insights-runtime-extractor-9v8g2\" (UID: \"466f60c6-0f78-4017-a90f-87489f71021c\") " pod="openshift-insights/insights-runtime-extractor-9v8g2"
Jul 30 00:18:30.303467 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.303271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/466f60c6-0f78-4017-a90f-87489f71021c-data-volume\") pod \"insights-runtime-extractor-9v8g2\" (UID: \"466f60c6-0f78-4017-a90f-87489f71021c\") " pod="openshift-insights/insights-runtime-extractor-9v8g2"
Jul 30 00:18:30.303467 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.303296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/466f60c6-0f78-4017-a90f-87489f71021c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9v8g2\" (UID: \"466f60c6-0f78-4017-a90f-87489f71021c\") " pod="openshift-insights/insights-runtime-extractor-9v8g2"
Jul 30 00:18:30.303467 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.303320    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/466f60c6-0f78-4017-a90f-87489f71021c-crio-socket\") pod \"insights-runtime-extractor-9v8g2\" (UID: \"466f60c6-0f78-4017-a90f-87489f71021c\") " pod="openshift-insights/insights-runtime-extractor-9v8g2"
Jul 30 00:18:30.303467 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.303341    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b39fd448-9f15-4275-86cf-5d233023c5a5-tmp-dir\") pod \"dns-default-lcr97\" (UID: \"b39fd448-9f15-4275-86cf-5d233023c5a5\") " pod="openshift-dns/dns-default-lcr97"
Jul 30 00:18:30.303467 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.303366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c0002be-c29e-4574-9d66-739390a6a6d3-cert\") pod \"ingress-canary-7pwht\" (UID: \"2c0002be-c29e-4574-9d66-739390a6a6d3\") " pod="openshift-ingress-canary/ingress-canary-7pwht"
Jul 30 00:18:30.303467 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.303388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkzr\" (UniqueName: \"kubernetes.io/projected/2c0002be-c29e-4574-9d66-739390a6a6d3-kube-api-access-nzkzr\") pod \"ingress-canary-7pwht\" (UID: \"2c0002be-c29e-4574-9d66-739390a6a6d3\") " pod="openshift-ingress-canary/ingress-canary-7pwht"
Jul 30 00:18:30.303751 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.303717    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/466f60c6-0f78-4017-a90f-87489f71021c-crio-socket\") pod \"insights-runtime-extractor-9v8g2\" (UID: \"466f60c6-0f78-4017-a90f-87489f71021c\") " pod="openshift-insights/insights-runtime-extractor-9v8g2"
Jul 30 00:18:30.303843 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.303815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b39fd448-9f15-4275-86cf-5d233023c5a5-config-volume\") pod \"dns-default-lcr97\" (UID: \"b39fd448-9f15-4275-86cf-5d233023c5a5\") " pod="openshift-dns/dns-default-lcr97"
Jul 30 00:18:30.303902 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.303841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b39fd448-9f15-4275-86cf-5d233023c5a5-tmp-dir\") pod \"dns-default-lcr97\" (UID: \"b39fd448-9f15-4275-86cf-5d233023c5a5\") " pod="openshift-dns/dns-default-lcr97"
Jul 30 00:18:30.304024 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.304001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/466f60c6-0f78-4017-a90f-87489f71021c-data-volume\") pod \"insights-runtime-extractor-9v8g2\" (UID: \"466f60c6-0f78-4017-a90f-87489f71021c\") " pod="openshift-insights/insights-runtime-extractor-9v8g2"
Jul 30 00:18:30.304322 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.304299    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/466f60c6-0f78-4017-a90f-87489f71021c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9v8g2\" (UID: \"466f60c6-0f78-4017-a90f-87489f71021c\") " pod="openshift-insights/insights-runtime-extractor-9v8g2"
Jul 30 00:18:30.307476 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.307451    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39fd448-9f15-4275-86cf-5d233023c5a5-metrics-tls\") pod \"dns-default-lcr97\" (UID: \"b39fd448-9f15-4275-86cf-5d233023c5a5\") " pod="openshift-dns/dns-default-lcr97"
Jul 30 00:18:30.307476 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.307469    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/466f60c6-0f78-4017-a90f-87489f71021c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9v8g2\" (UID: \"466f60c6-0f78-4017-a90f-87489f71021c\") " pod="openshift-insights/insights-runtime-extractor-9v8g2"
Jul 30 00:18:30.307591 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.307512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c0002be-c29e-4574-9d66-739390a6a6d3-cert\") pod \"ingress-canary-7pwht\" (UID: \"2c0002be-c29e-4574-9d66-739390a6a6d3\") " pod="openshift-ingress-canary/ingress-canary-7pwht"
Jul 30 00:18:30.311259 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.311216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79gwx\" (UniqueName: \"kubernetes.io/projected/466f60c6-0f78-4017-a90f-87489f71021c-kube-api-access-79gwx\") pod \"insights-runtime-extractor-9v8g2\" (UID: \"466f60c6-0f78-4017-a90f-87489f71021c\") " pod="openshift-insights/insights-runtime-extractor-9v8g2"
Jul 30 00:18:30.311574 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.311550    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br7r5\" (UniqueName: \"kubernetes.io/projected/b39fd448-9f15-4275-86cf-5d233023c5a5-kube-api-access-br7r5\") pod \"dns-default-lcr97\" (UID: \"b39fd448-9f15-4275-86cf-5d233023c5a5\") " pod="openshift-dns/dns-default-lcr97"
Jul 30 00:18:30.311724 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.311711    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzkzr\" (UniqueName: \"kubernetes.io/projected/2c0002be-c29e-4574-9d66-739390a6a6d3-kube-api-access-nzkzr\") pod \"ingress-canary-7pwht\" (UID: \"2c0002be-c29e-4574-9d66-739390a6a6d3\") " pod="openshift-ingress-canary/ingress-canary-7pwht"
Jul 30 00:18:30.316525 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.316505    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:18:30.316600 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.316531    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:18:30.320592 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.320572    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f7pzr\""
Jul 30 00:18:30.320592 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.320589    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qvw7x\""
Jul 30 00:18:30.321033 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.320594    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 30 00:18:30.321033 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.320572    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 30 00:18:30.321033 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.320572    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 30 00:18:30.342168 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.342148    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7pwht"
Jul 30 00:18:30.357335 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.357317    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9v8g2"
Jul 30 00:18:30.374552 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.374528    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lcr97"
Jul 30 00:18:30.504718 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.504350    2579 generic.go:356] "Generic (PLEG): container finished" podID="7f6d919d-802a-4eb8-a7ec-948626dfc26b" containerID="f41795d7a5803aafdd5604eda282a4cc80d62d45728798a1b396c3837363ef1c" exitCode=0
Jul 30 00:18:30.504718 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.504622    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfkf9" event={"ID":"7f6d919d-802a-4eb8-a7ec-948626dfc26b","Type":"ContainerDied","Data":"f41795d7a5803aafdd5604eda282a4cc80d62d45728798a1b396c3837363ef1c"}
Jul 30 00:18:30.540686 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.540650    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7pwht"]
Jul 30 00:18:30.545174 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:18:30.545139    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0002be_c29e_4574_9d66_739390a6a6d3.slice/crio-80d7e42493b6d1701fea5f7be77a421e2cff608069bfe9291abe1a3e84630729 WatchSource:0}: Error finding container 80d7e42493b6d1701fea5f7be77a421e2cff608069bfe9291abe1a3e84630729: Status 404 returned error can't find the container with id 80d7e42493b6d1701fea5f7be77a421e2cff608069bfe9291abe1a3e84630729
Jul 30 00:18:30.546944 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.546914    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9v8g2"]
Jul 30 00:18:30.550189 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:18:30.550161    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466f60c6_0f78_4017_a90f_87489f71021c.slice/crio-b89603a18016c395888d26fc4491625cb49c30b099ad7a88fbf5a7bc451cbdbc WatchSource:0}: Error finding container b89603a18016c395888d26fc4491625cb49c30b099ad7a88fbf5a7bc451cbdbc: Status 404 returned error can't find the container with id b89603a18016c395888d26fc4491625cb49c30b099ad7a88fbf5a7bc451cbdbc
Jul 30 00:18:30.552152 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:30.552089    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lcr97"]
Jul 30 00:18:30.558679 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:18:30.558648    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39fd448_9f15_4275_86cf_5d233023c5a5.slice/crio-c030cf40b6f0c2e20beaaa0e0611dd427efc90e2e963cfe1a88dfdde55a080d8 WatchSource:0}: Error finding container c030cf40b6f0c2e20beaaa0e0611dd427efc90e2e963cfe1a88dfdde55a080d8: Status 404 returned error can't find the container with id c030cf40b6f0c2e20beaaa0e0611dd427efc90e2e963cfe1a88dfdde55a080d8
Jul 30 00:18:31.322253 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:31.322206    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:18:31.325337 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:31.325107    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 30 00:18:31.513349 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:31.513307    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfkf9" event={"ID":"7f6d919d-802a-4eb8-a7ec-948626dfc26b","Type":"ContainerStarted","Data":"c479b1803ae41b8ffd3919ffbf5b8d49a59d459ba623224d719e88906fee9753"}
Jul 30 00:18:31.516551 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:31.516519    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lcr97" event={"ID":"b39fd448-9f15-4275-86cf-5d233023c5a5","Type":"ContainerStarted","Data":"c030cf40b6f0c2e20beaaa0e0611dd427efc90e2e963cfe1a88dfdde55a080d8"}
Jul 30 00:18:31.518194 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:31.518164    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9v8g2" event={"ID":"466f60c6-0f78-4017-a90f-87489f71021c","Type":"ContainerStarted","Data":"b89603a18016c395888d26fc4491625cb49c30b099ad7a88fbf5a7bc451cbdbc"}
Jul 30 00:18:31.520690 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:31.520632    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7pwht" event={"ID":"2c0002be-c29e-4574-9d66-739390a6a6d3","Type":"ContainerStarted","Data":"80d7e42493b6d1701fea5f7be77a421e2cff608069bfe9291abe1a3e84630729"}
Jul 30 00:18:31.537207 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:31.537151    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jfkf9" podStartSLOduration=7.527094454 podStartE2EDuration="36.537132696s" totalImagesPullingTime="29.010038242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:31.53592562 +0000 UTC m=+36.833192559" watchObservedRunningTime="2026-07-30 00:18:31.537132696 +0000 UTC m=+36.834399628"
Jul 30 00:18:34.099480 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.099446    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-597b49ddf8-6blvn"]
Jul 30 00:18:34.144557 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.144522    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-597b49ddf8-6blvn"]
Jul 30 00:18:34.144722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.144632    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:34.147847 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.147763    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 30 00:18:34.148072 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.148055    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 30 00:18:34.148975 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.148951    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 30 00:18:34.149090 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.148975    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pfswq\""
Jul 30 00:18:34.149090 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.148986    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 30 00:18:34.149283 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.149092    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 30 00:18:34.149283 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.149186    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 30 00:18:34.149420 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.149292    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 30 00:18:34.243198 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.243132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qqs9\" (UniqueName: \"kubernetes.io/projected/c855f937-8419-4e43-ab83-46aae0c940b5-kube-api-access-8qqs9\") pod \"console-597b49ddf8-6blvn\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") " pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:34.243198 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.243177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c855f937-8419-4e43-ab83-46aae0c940b5-oauth-serving-cert\") pod \"console-597b49ddf8-6blvn\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") " pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:34.243441 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.243263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c855f937-8419-4e43-ab83-46aae0c940b5-console-oauth-config\") pod \"console-597b49ddf8-6blvn\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") " pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:34.243441 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.243288    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c855f937-8419-4e43-ab83-46aae0c940b5-console-config\") pod \"console-597b49ddf8-6blvn\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") " pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:34.243441 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.243315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c855f937-8419-4e43-ab83-46aae0c940b5-console-serving-cert\") pod \"console-597b49ddf8-6blvn\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") " pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:34.243441 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.243408    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c855f937-8419-4e43-ab83-46aae0c940b5-service-ca\") pod \"console-597b49ddf8-6blvn\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") " pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:34.345034 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.345001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c855f937-8419-4e43-ab83-46aae0c940b5-service-ca\") pod \"console-597b49ddf8-6blvn\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") " pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:34.345228 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.345045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qqs9\" (UniqueName: \"kubernetes.io/projected/c855f937-8419-4e43-ab83-46aae0c940b5-kube-api-access-8qqs9\") pod \"console-597b49ddf8-6blvn\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") " pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:34.345228 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.345071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c855f937-8419-4e43-ab83-46aae0c940b5-oauth-serving-cert\") pod \"console-597b49ddf8-6blvn\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") " pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:34.345228 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.345140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c855f937-8419-4e43-ab83-46aae0c940b5-console-oauth-config\") pod \"console-597b49ddf8-6blvn\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") " pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:34.345228 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.345162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c855f937-8419-4e43-ab83-46aae0c940b5-console-config\") pod \"console-597b49ddf8-6blvn\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") " pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:34.345228 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.345186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c855f937-8419-4e43-ab83-46aae0c940b5-console-serving-cert\") pod \"console-597b49ddf8-6blvn\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") " pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:34.345890 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.345864    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c855f937-8419-4e43-ab83-46aae0c940b5-service-ca\") pod \"console-597b49ddf8-6blvn\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") " pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:34.346322 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.346292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c855f937-8419-4e43-ab83-46aae0c940b5-oauth-serving-cert\") pod \"console-597b49ddf8-6blvn\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") " pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:34.346409 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.346318    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c855f937-8419-4e43-ab83-46aae0c940b5-console-config\") pod \"console-597b49ddf8-6blvn\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") " pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:34.349633 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.349500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c855f937-8419-4e43-ab83-46aae0c940b5-console-oauth-config\") pod \"console-597b49ddf8-6blvn\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") " pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:34.349633 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.349509    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c855f937-8419-4e43-ab83-46aae0c940b5-console-serving-cert\") pod \"console-597b49ddf8-6blvn\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") " pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:34.353570 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.353551    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qqs9\" (UniqueName: \"kubernetes.io/projected/c855f937-8419-4e43-ab83-46aae0c940b5-kube-api-access-8qqs9\") pod \"console-597b49ddf8-6blvn\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") " pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:34.456733 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:34.456697    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:37.335585 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:37.335552    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-597b49ddf8-6blvn"]
Jul 30 00:18:37.338309 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:18:37.338284    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc855f937_8419_4e43_ab83_46aae0c940b5.slice/crio-7be5b007aaad025dd9ce9b1a980da2a91410c313746219a95c85eb93ae80de44 WatchSource:0}: Error finding container 7be5b007aaad025dd9ce9b1a980da2a91410c313746219a95c85eb93ae80de44: Status 404 returned error can't find the container with id 7be5b007aaad025dd9ce9b1a980da2a91410c313746219a95c85eb93ae80de44
Jul 30 00:18:37.539554 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:37.539517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7pwht" event={"ID":"2c0002be-c29e-4574-9d66-739390a6a6d3","Type":"ContainerStarted","Data":"392f301b2993b5b0d492cd3137f38506edd1efe5037f9e66942af6c1874d6f98"}
Jul 30 00:18:37.540937 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:37.540905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597b49ddf8-6blvn" event={"ID":"c855f937-8419-4e43-ab83-46aae0c940b5","Type":"ContainerStarted","Data":"7be5b007aaad025dd9ce9b1a980da2a91410c313746219a95c85eb93ae80de44"}
Jul 30 00:18:37.542703 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:37.542652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lcr97" event={"ID":"b39fd448-9f15-4275-86cf-5d233023c5a5","Type":"ContainerStarted","Data":"888b4101b89b733045125d8d9fab6b0cc818748fdbebc73cfdfe400c4efa6e91"}
Jul 30 00:18:37.542703 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:37.542696    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lcr97" event={"ID":"b39fd448-9f15-4275-86cf-5d233023c5a5","Type":"ContainerStarted","Data":"6113a39155d11974ac363749ef8d1b58744c5d47531299a546cce3381fdf414f"}
Jul 30 00:18:37.542896 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:37.542797    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lcr97"
Jul 30 00:18:37.554968 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:37.554930    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7pwht" podStartSLOduration=1.926422807 podStartE2EDuration="8.554917194s" totalImagesPullingTime="6.628494387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:37.554269115 +0000 UTC m=+42.851536047" watchObservedRunningTime="2026-07-30 00:18:37.554917194 +0000 UTC m=+42.852184122"
Jul 30 00:18:37.571095 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:37.571052    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lcr97" podStartSLOduration=0.957773446 podStartE2EDuration="7.571038944s" totalImagesPullingTime="6.613265498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:37.57077782 +0000 UTC m=+42.868044752" watchObservedRunningTime="2026-07-30 00:18:37.571038944 +0000 UTC m=+42.868305876"
Jul 30 00:18:38.548000 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:38.547931    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9v8g2" event={"ID":"466f60c6-0f78-4017-a90f-87489f71021c","Type":"ContainerStarted","Data":"c93c743278adaf94e24ce33110c6afc68cf2ecb59051048dba77dc0848366309"}
Jul 30 00:18:40.245095 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.245054    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-7j8rn"]
Jul 30 00:18:40.293763 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.293722    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-7j8rn"]
Jul 30 00:18:40.293951 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.293861    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-7j8rn"
Jul 30 00:18:40.297120 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.297097    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 30 00:18:40.298788 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.298484    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 30 00:18:40.298788 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.298513    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 30 00:18:40.298788 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.298582    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 30 00:18:40.298788 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.298582    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7vbth\""
Jul 30 00:18:40.298788 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.298660    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 30 00:18:40.394637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.394604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28d39885-3cc1-4f91-80d1-b8d0cd31bef1-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-7j8rn\" (UID: \"28d39885-3cc1-4f91-80d1-b8d0cd31bef1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7j8rn"
Jul 30 00:18:40.394637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.394640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzqb2\" (UniqueName: \"kubernetes.io/projected/28d39885-3cc1-4f91-80d1-b8d0cd31bef1-kube-api-access-rzqb2\") pod \"prometheus-operator-84d69f794d-7j8rn\" (UID: \"28d39885-3cc1-4f91-80d1-b8d0cd31bef1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7j8rn"
Jul 30 00:18:40.394951 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.394791    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/28d39885-3cc1-4f91-80d1-b8d0cd31bef1-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-7j8rn\" (UID: \"28d39885-3cc1-4f91-80d1-b8d0cd31bef1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7j8rn"
Jul 30 00:18:40.394951 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.394828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28d39885-3cc1-4f91-80d1-b8d0cd31bef1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-7j8rn\" (UID: \"28d39885-3cc1-4f91-80d1-b8d0cd31bef1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7j8rn"
Jul 30 00:18:40.496067 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.495969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28d39885-3cc1-4f91-80d1-b8d0cd31bef1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-7j8rn\" (UID: \"28d39885-3cc1-4f91-80d1-b8d0cd31bef1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7j8rn"
Jul 30 00:18:40.496067 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.496048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28d39885-3cc1-4f91-80d1-b8d0cd31bef1-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-7j8rn\" (UID: \"28d39885-3cc1-4f91-80d1-b8d0cd31bef1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7j8rn"
Jul 30 00:18:40.496303 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.496071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzqb2\" (UniqueName: \"kubernetes.io/projected/28d39885-3cc1-4f91-80d1-b8d0cd31bef1-kube-api-access-rzqb2\") pod \"prometheus-operator-84d69f794d-7j8rn\" (UID: \"28d39885-3cc1-4f91-80d1-b8d0cd31bef1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7j8rn"
Jul 30 00:18:40.496303 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.496152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/28d39885-3cc1-4f91-80d1-b8d0cd31bef1-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-7j8rn\" (UID: \"28d39885-3cc1-4f91-80d1-b8d0cd31bef1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7j8rn"
Jul 30 00:18:40.496912 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.496883    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28d39885-3cc1-4f91-80d1-b8d0cd31bef1-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-7j8rn\" (UID: \"28d39885-3cc1-4f91-80d1-b8d0cd31bef1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7j8rn"
Jul 30 00:18:40.499004 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.498979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/28d39885-3cc1-4f91-80d1-b8d0cd31bef1-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-7j8rn\" (UID: \"28d39885-3cc1-4f91-80d1-b8d0cd31bef1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7j8rn"
Jul 30 00:18:40.499188 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.499162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28d39885-3cc1-4f91-80d1-b8d0cd31bef1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-7j8rn\" (UID: \"28d39885-3cc1-4f91-80d1-b8d0cd31bef1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7j8rn"
Jul 30 00:18:40.504151 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.504122    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzqb2\" (UniqueName: \"kubernetes.io/projected/28d39885-3cc1-4f91-80d1-b8d0cd31bef1-kube-api-access-rzqb2\") pod \"prometheus-operator-84d69f794d-7j8rn\" (UID: \"28d39885-3cc1-4f91-80d1-b8d0cd31bef1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7j8rn"
Jul 30 00:18:40.606471 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:40.606441    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-7j8rn"
Jul 30 00:18:43.421177 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:43.421143    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-7j8rn"]
Jul 30 00:18:43.424410 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:18:43.424377    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d39885_3cc1_4f91_80d1_b8d0cd31bef1.slice/crio-12473362afae643a4747e8ed112c8553752d41385c14d71ede20ba73b28cb38c WatchSource:0}: Error finding container 12473362afae643a4747e8ed112c8553752d41385c14d71ede20ba73b28cb38c: Status 404 returned error can't find the container with id 12473362afae643a4747e8ed112c8553752d41385c14d71ede20ba73b28cb38c
Jul 30 00:18:43.564524 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:43.564426    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-7j8rn" event={"ID":"28d39885-3cc1-4f91-80d1-b8d0cd31bef1","Type":"ContainerStarted","Data":"12473362afae643a4747e8ed112c8553752d41385c14d71ede20ba73b28cb38c"}
Jul 30 00:18:43.565912 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:43.565886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597b49ddf8-6blvn" event={"ID":"c855f937-8419-4e43-ab83-46aae0c940b5","Type":"ContainerStarted","Data":"cdfe8ecb2ca7a51cc5e8e1af307db18641c10dcc1f67601c5e1a5af4d937c2a9"}
Jul 30 00:18:43.582400 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:43.582352    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-597b49ddf8-6blvn" podStartSLOduration=3.62470715 podStartE2EDuration="9.582335487s" totalImagesPullingTime="5.957628337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:43.581559557 +0000 UTC m=+48.878826489" watchObservedRunningTime="2026-07-30 00:18:43.582335487 +0000 UTC m=+48.879602419"
Jul 30 00:18:44.457762 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:44.457714    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:44.458215 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:44.457832    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:18:44.459357 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:44.459323    2579 patch_prober.go:28] interesting pod/console-597b49ddf8-6blvn container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" start-of-body=
Jul 30 00:18:44.459493 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:44.459402    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-597b49ddf8-6blvn" podUID="c855f937-8419-4e43-ab83-46aae0c940b5" containerName="console" probeResult="failure" output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused"
Jul 30 00:18:44.573750 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:44.573702    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9v8g2" event={"ID":"466f60c6-0f78-4017-a90f-87489f71021c","Type":"ContainerStarted","Data":"c1192fd55f597627e37e837391ab76b4341b025ce3252ef1c5e7b7dda2ab08f1"}
Jul 30 00:18:45.579531 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:45.579392    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-7j8rn" event={"ID":"28d39885-3cc1-4f91-80d1-b8d0cd31bef1","Type":"ContainerStarted","Data":"31e6ab695d779d5814058373b8c130536c50e8742489d14ad6ca77b5cfcf6008"}
Jul 30 00:18:45.579531 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:45.579452    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-7j8rn" event={"ID":"28d39885-3cc1-4f91-80d1-b8d0cd31bef1","Type":"ContainerStarted","Data":"1dd2c50db72c0af9277ffaee9add99d58a49bc35029d65571e25f0a6d8ce18d5"}
Jul 30 00:18:45.599223 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:45.599166    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-7j8rn" podStartSLOduration=3.733652696 podStartE2EDuration="5.59914916s" totalImagesPullingTime="1.865496464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:45.59690447 +0000 UTC m=+50.894171402" watchObservedRunningTime="2026-07-30 00:18:45.59914916 +0000 UTC m=+50.896416092"
Jul 30 00:18:47.550156 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.550123    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lcr97"
Jul 30 00:18:47.588266 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.588216    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lf4z8"]
Jul 30 00:18:47.620610 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.620581    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.623209 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.623181    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 30 00:18:47.623387 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.623184    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 30 00:18:47.623387 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.623322    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b9dj7\""
Jul 30 00:18:47.623503 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.623480    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 30 00:18:47.651326 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.651304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d7a41fca-3a9c-469a-ad8a-79047736d209-root\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.651456 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.651351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d7a41fca-3a9c-469a-ad8a-79047736d209-node-exporter-textfile\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.651456 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.651423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7a41fca-3a9c-469a-ad8a-79047736d209-node-exporter-tls\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.651569 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.651466    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7a41fca-3a9c-469a-ad8a-79047736d209-metrics-client-ca\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.651569 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.651503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krx7b\" (UniqueName: \"kubernetes.io/projected/d7a41fca-3a9c-469a-ad8a-79047736d209-kube-api-access-krx7b\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.651637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.651575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d7a41fca-3a9c-469a-ad8a-79047736d209-node-exporter-accelerators-collector-config\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.651671 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.651656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7a41fca-3a9c-469a-ad8a-79047736d209-sys\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.651706 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.651681    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7a41fca-3a9c-469a-ad8a-79047736d209-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.651765 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.651751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d7a41fca-3a9c-469a-ad8a-79047736d209-node-exporter-wtmp\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.752728 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.752698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d7a41fca-3a9c-469a-ad8a-79047736d209-node-exporter-wtmp\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.752838 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.752745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d7a41fca-3a9c-469a-ad8a-79047736d209-root\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.752838 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.752798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d7a41fca-3a9c-469a-ad8a-79047736d209-node-exporter-textfile\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.752951 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.752846    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d7a41fca-3a9c-469a-ad8a-79047736d209-root\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.752951 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.752850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7a41fca-3a9c-469a-ad8a-79047736d209-node-exporter-tls\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.752951 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.752901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7a41fca-3a9c-469a-ad8a-79047736d209-metrics-client-ca\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.752951 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.752925    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krx7b\" (UniqueName: \"kubernetes.io/projected/d7a41fca-3a9c-469a-ad8a-79047736d209-kube-api-access-krx7b\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.753196 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.752849    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d7a41fca-3a9c-469a-ad8a-79047736d209-node-exporter-wtmp\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.753196 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:47.752958    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 30 00:18:47.753196 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:47.753037    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a41fca-3a9c-469a-ad8a-79047736d209-node-exporter-tls podName:d7a41fca-3a9c-469a-ad8a-79047736d209 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:48.252998903 +0000 UTC m=+53.550265825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d7a41fca-3a9c-469a-ad8a-79047736d209-node-exporter-tls") pod "node-exporter-lf4z8" (UID: "d7a41fca-3a9c-469a-ad8a-79047736d209") : secret "node-exporter-tls" not found
Jul 30 00:18:47.753196 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.752950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d7a41fca-3a9c-469a-ad8a-79047736d209-node-exporter-accelerators-collector-config\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.753196 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.753156    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d7a41fca-3a9c-469a-ad8a-79047736d209-node-exporter-textfile\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.753196 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.753170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7a41fca-3a9c-469a-ad8a-79047736d209-sys\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.753519 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.753214    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7a41fca-3a9c-469a-ad8a-79047736d209-sys\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.753519 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.753215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7a41fca-3a9c-469a-ad8a-79047736d209-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.753519 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.753482    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d7a41fca-3a9c-469a-ad8a-79047736d209-node-exporter-accelerators-collector-config\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.753519 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.753506    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7a41fca-3a9c-469a-ad8a-79047736d209-metrics-client-ca\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.755404 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.755384    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7a41fca-3a9c-469a-ad8a-79047736d209-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:47.761483 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:47.761462    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krx7b\" (UniqueName: \"kubernetes.io/projected/d7a41fca-3a9c-469a-ad8a-79047736d209-kube-api-access-krx7b\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:48.257481 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.257446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7a41fca-3a9c-469a-ad8a-79047736d209-node-exporter-tls\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:48.259882 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.259858    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7a41fca-3a9c-469a-ad8a-79047736d209-node-exporter-tls\") pod \"node-exporter-lf4z8\" (UID: \"d7a41fca-3a9c-469a-ad8a-79047736d209\") " pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:48.530398 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.530316    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lf4z8"
Jul 30 00:18:48.540398 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:18:48.540367    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a41fca_3a9c_469a_ad8a_79047736d209.slice/crio-2c224589a0a38eab84270cec86bb24ba784101466c95b1229d13189045b3a457 WatchSource:0}: Error finding container 2c224589a0a38eab84270cec86bb24ba784101466c95b1229d13189045b3a457: Status 404 returned error can't find the container with id 2c224589a0a38eab84270cec86bb24ba784101466c95b1229d13189045b3a457
Jul 30 00:18:48.591726 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.591686    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9v8g2" event={"ID":"466f60c6-0f78-4017-a90f-87489f71021c","Type":"ContainerStarted","Data":"1f9d3bb6e5cf24ed381e3b4f7d3b8d6ed80c3fa2a5576c0c0b43a16689f31eb5"}
Jul 30 00:18:48.592828 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.592806    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lf4z8" event={"ID":"d7a41fca-3a9c-469a-ad8a-79047736d209","Type":"ContainerStarted","Data":"2c224589a0a38eab84270cec86bb24ba784101466c95b1229d13189045b3a457"}
Jul 30 00:18:48.613846 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.613802    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9v8g2" podStartSLOduration=2.634995104 podStartE2EDuration="19.613785256s" totalImagesPullingTime="16.978790152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:18:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:48.613451113 +0000 UTC m=+53.910718044" watchObservedRunningTime="2026-07-30 00:18:48.613785256 +0000 UTC m=+53.911052181"
Jul 30 00:18:48.662292 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.662263    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 00:18:48.666290 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.666268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.669169 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.669147    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 30 00:18:48.669294 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.669195    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 30 00:18:48.669294 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.669211    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 30 00:18:48.669462 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.669449    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 30 00:18:48.669526 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.669487    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 30 00:18:48.669777 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.669755    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 30 00:18:48.669777 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.669763    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-j4gt8\""
Jul 30 00:18:48.669904 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.669816    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 30 00:18:48.669904 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.669762    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 30 00:18:48.669904 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.669820    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 30 00:18:48.682626 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.682604    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 00:18:48.761785 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.761759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/52138b7c-2f87-486d-855c-dc6bb4c8c39b-config-out\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.761929 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.761870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.761929 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.761918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52138b7c-2f87-486d-855c-dc6bb4c8c39b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.762007 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.761941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-config-volume\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.762007 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.761969    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/52138b7c-2f87-486d-855c-dc6bb4c8c39b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.762007 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.762002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.762104 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.762018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/52138b7c-2f87-486d-855c-dc6bb4c8c39b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.762156 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.762106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.762193 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.762170    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.762237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.762189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7fbj\" (UniqueName: \"kubernetes.io/projected/52138b7c-2f87-486d-855c-dc6bb4c8c39b-kube-api-access-m7fbj\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.762237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.762225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.762335 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.762256    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52138b7c-2f87-486d-855c-dc6bb4c8c39b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.762335 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.762283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-web-config\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.863178 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.863094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.863178 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.863129    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52138b7c-2f87-486d-855c-dc6bb4c8c39b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.863178 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.863152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-config-volume\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.863178 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.863173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/52138b7c-2f87-486d-855c-dc6bb4c8c39b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.863957 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.863200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.863957 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.863217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/52138b7c-2f87-486d-855c-dc6bb4c8c39b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.863957 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.863281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.863957 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.863343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.863957 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.863370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fbj\" (UniqueName: \"kubernetes.io/projected/52138b7c-2f87-486d-855c-dc6bb4c8c39b-kube-api-access-m7fbj\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.863957 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.863410    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.863957 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.863569    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52138b7c-2f87-486d-855c-dc6bb4c8c39b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.863957 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.863606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-web-config\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.863957 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.863647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/52138b7c-2f87-486d-855c-dc6bb4c8c39b-config-out\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.865760 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.865729    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/52138b7c-2f87-486d-855c-dc6bb4c8c39b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.866075 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.866049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52138b7c-2f87-486d-855c-dc6bb4c8c39b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.866781 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.866587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.866781 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.866609    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/52138b7c-2f87-486d-855c-dc6bb4c8c39b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.867842 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.867783    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52138b7c-2f87-486d-855c-dc6bb4c8c39b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.869271 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.868343    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.869271 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.868532    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.869271 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.868754    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.869766 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.869743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.871660 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.871638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-config-volume\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.872319 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.872297    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/52138b7c-2f87-486d-855c-dc6bb4c8c39b-config-out\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.872416 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.872405    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-web-config\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.874508 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.874489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7fbj\" (UniqueName: \"kubernetes.io/projected/52138b7c-2f87-486d-855c-dc6bb4c8c39b-kube-api-access-m7fbj\") pod \"alertmanager-main-0\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:48.976464 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:48.976432    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:18:49.113267 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.113179    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 00:18:49.114309 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:18:49.114275    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52138b7c_2f87_486d_855c_dc6bb4c8c39b.slice/crio-2fea0dd37cb6fc37e387e1a8660b8c9456a0102077de34ef2f7122df2d40025c WatchSource:0}: Error finding container 2fea0dd37cb6fc37e387e1a8660b8c9456a0102077de34ef2f7122df2d40025c: Status 404 returned error can't find the container with id 2fea0dd37cb6fc37e387e1a8660b8c9456a0102077de34ef2f7122df2d40025c
Jul 30 00:18:49.597128 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.597097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52138b7c-2f87-486d-855c-dc6bb4c8c39b","Type":"ContainerStarted","Data":"2fea0dd37cb6fc37e387e1a8660b8c9456a0102077de34ef2f7122df2d40025c"}
Jul 30 00:18:49.673779 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.673752    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-b799ccf8b-62djb"]
Jul 30 00:18:49.676217 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.676189    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.678931 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.678906    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 30 00:18:49.679056 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.678962    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 30 00:18:49.679056 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.679034    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 30 00:18:49.679175 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.679087    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4mt9p\""
Jul 30 00:18:49.679175 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.679141    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 30 00:18:49.679319 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.679206    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 30 00:18:49.680127 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.680110    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-345u55ak6sknn\""
Jul 30 00:18:49.688636 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.688614    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b799ccf8b-62djb"]
Jul 30 00:18:49.773523 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.773491    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3aa3fcdd-0019-4f8c-a799-6509d727d472-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.773661 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.773528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kb4h\" (UniqueName: \"kubernetes.io/projected/3aa3fcdd-0019-4f8c-a799-6509d727d472-kube-api-access-6kb4h\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.773661 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.773568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3aa3fcdd-0019-4f8c-a799-6509d727d472-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.773661 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.773597    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3aa3fcdd-0019-4f8c-a799-6509d727d472-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.773661 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.773643    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3aa3fcdd-0019-4f8c-a799-6509d727d472-metrics-client-ca\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.773825 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.773684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3aa3fcdd-0019-4f8c-a799-6509d727d472-secret-thanos-querier-tls\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.773825 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.773745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3aa3fcdd-0019-4f8c-a799-6509d727d472-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.773825 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.773770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3aa3fcdd-0019-4f8c-a799-6509d727d472-secret-grpc-tls\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.874598 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.874572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3aa3fcdd-0019-4f8c-a799-6509d727d472-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.874686 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.874603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3aa3fcdd-0019-4f8c-a799-6509d727d472-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.874686 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.874663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3aa3fcdd-0019-4f8c-a799-6509d727d472-metrics-client-ca\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.874764 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.874708    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3aa3fcdd-0019-4f8c-a799-6509d727d472-secret-thanos-querier-tls\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.874764 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.874742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3aa3fcdd-0019-4f8c-a799-6509d727d472-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.874764 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.874761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3aa3fcdd-0019-4f8c-a799-6509d727d472-secret-grpc-tls\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.874909 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.874811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3aa3fcdd-0019-4f8c-a799-6509d727d472-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.874909 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.874840    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kb4h\" (UniqueName: \"kubernetes.io/projected/3aa3fcdd-0019-4f8c-a799-6509d727d472-kube-api-access-6kb4h\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.875546 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.875520    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3aa3fcdd-0019-4f8c-a799-6509d727d472-metrics-client-ca\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.877290 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.877235    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3aa3fcdd-0019-4f8c-a799-6509d727d472-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.877404 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.877341    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3aa3fcdd-0019-4f8c-a799-6509d727d472-secret-thanos-querier-tls\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.877518 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.877499    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3aa3fcdd-0019-4f8c-a799-6509d727d472-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.877583 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.877562    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3aa3fcdd-0019-4f8c-a799-6509d727d472-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.877725 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.877704    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3aa3fcdd-0019-4f8c-a799-6509d727d472-secret-grpc-tls\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.877912 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.877893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3aa3fcdd-0019-4f8c-a799-6509d727d472-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.883067 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.883043    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kb4h\" (UniqueName: \"kubernetes.io/projected/3aa3fcdd-0019-4f8c-a799-6509d727d472-kube-api-access-6kb4h\") pod \"thanos-querier-b799ccf8b-62djb\" (UID: \"3aa3fcdd-0019-4f8c-a799-6509d727d472\") " pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:49.986165 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:49.986138    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:50.130717 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:50.130590    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b799ccf8b-62djb"]
Jul 30 00:18:50.133201 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:18:50.133173    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa3fcdd_0019_4f8c_a799_6509d727d472.slice/crio-7c4ef82efe13ff71de84c64148d15b7c57049ab08c3f751b3e4f7be6b00c67f2 WatchSource:0}: Error finding container 7c4ef82efe13ff71de84c64148d15b7c57049ab08c3f751b3e4f7be6b00c67f2: Status 404 returned error can't find the container with id 7c4ef82efe13ff71de84c64148d15b7c57049ab08c3f751b3e4f7be6b00c67f2
Jul 30 00:18:50.601723 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:50.601679    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb" event={"ID":"3aa3fcdd-0019-4f8c-a799-6509d727d472","Type":"ContainerStarted","Data":"7c4ef82efe13ff71de84c64148d15b7c57049ab08c3f751b3e4f7be6b00c67f2"}
Jul 30 00:18:50.605332 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:50.605299    2579 generic.go:356] "Generic (PLEG): container finished" podID="d7a41fca-3a9c-469a-ad8a-79047736d209" containerID="a39f0534e5f6b3ffa20a77cda48da0bbd339b883305fb309472ec580baf4f879" exitCode=0
Jul 30 00:18:50.605441 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:50.605345    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lf4z8" event={"ID":"d7a41fca-3a9c-469a-ad8a-79047736d209","Type":"ContainerDied","Data":"a39f0534e5f6b3ffa20a77cda48da0bbd339b883305fb309472ec580baf4f879"}
Jul 30 00:18:51.616384 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:51.616342    2579 generic.go:356] "Generic (PLEG): container finished" podID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerID="2f140f96642b0c5aa1503411fe451d423c257dfd7ab1e1452506e8bd9f250661" exitCode=0
Jul 30 00:18:51.616811 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:51.616443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52138b7c-2f87-486d-855c-dc6bb4c8c39b","Type":"ContainerDied","Data":"2f140f96642b0c5aa1503411fe451d423c257dfd7ab1e1452506e8bd9f250661"}
Jul 30 00:18:51.620134 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:51.620108    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lf4z8" event={"ID":"d7a41fca-3a9c-469a-ad8a-79047736d209","Type":"ContainerStarted","Data":"00ad22a3d1b1dd762b90dcce30ce7fb8211ce720d620a494c66156f52d9de946"}
Jul 30 00:18:51.620268 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:51.620166    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lf4z8" event={"ID":"d7a41fca-3a9c-469a-ad8a-79047736d209","Type":"ContainerStarted","Data":"6a95fb04a49da32119fafe928d757b393c2b2f12feed7a244de1dd5c069a01c8"}
Jul 30 00:18:51.665282 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:51.665202    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lf4z8" podStartSLOduration=3.566460391 podStartE2EDuration="4.665183551s" totalImagesPullingTime="1.031268889s" totalInitContainerRuntime="67.454271ms" isStatefulPod=true podCreationTimestamp="2026-07-30 00:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:51.663996537 +0000 UTC m=+56.961263470" watchObservedRunningTime="2026-07-30 00:18:51.665183551 +0000 UTC m=+56.962450462"
Jul 30 00:18:51.988952 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:51.988919    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-75c58d4475-nwtmf"]
Jul 30 00:18:51.991283 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:51.991268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:51.994014 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:51.993984    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-g2dl7\""
Jul 30 00:18:51.994014 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:51.993995    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 30 00:18:51.994014 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:51.994016    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 30 00:18:51.995223 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:51.995201    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 30 00:18:51.995333 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:51.995205    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-95cf3qj17e5b6\""
Jul 30 00:18:51.995393 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:51.995341    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 30 00:18:52.001286 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.001264    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-75c58d4475-nwtmf"]
Jul 30 00:18:52.097724 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.097694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0de6fb0f-5338-4568-a3b6-15a05b99b434-client-ca-bundle\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.097878 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.097731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0de6fb0f-5338-4568-a3b6-15a05b99b434-metrics-server-audit-profiles\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.097878 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.097819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0de6fb0f-5338-4568-a3b6-15a05b99b434-secret-metrics-server-client-certs\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.097878 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.097851    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0de6fb0f-5338-4568-a3b6-15a05b99b434-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.097981 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.097961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0de6fb0f-5338-4568-a3b6-15a05b99b434-secret-metrics-server-tls\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.098031 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.098012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0de6fb0f-5338-4568-a3b6-15a05b99b434-audit-log\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.098074 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.098047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnzph\" (UniqueName: \"kubernetes.io/projected/0de6fb0f-5338-4568-a3b6-15a05b99b434-kube-api-access-qnzph\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.198807 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.198762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0de6fb0f-5338-4568-a3b6-15a05b99b434-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.199006 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.198837    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0de6fb0f-5338-4568-a3b6-15a05b99b434-secret-metrics-server-tls\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.199006 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.198867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0de6fb0f-5338-4568-a3b6-15a05b99b434-audit-log\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.199006 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.198884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnzph\" (UniqueName: \"kubernetes.io/projected/0de6fb0f-5338-4568-a3b6-15a05b99b434-kube-api-access-qnzph\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.199006 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.198974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0de6fb0f-5338-4568-a3b6-15a05b99b434-client-ca-bundle\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.199200 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.199025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0de6fb0f-5338-4568-a3b6-15a05b99b434-metrics-server-audit-profiles\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.199200 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.199107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0de6fb0f-5338-4568-a3b6-15a05b99b434-secret-metrics-server-client-certs\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.199319 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.199295    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0de6fb0f-5338-4568-a3b6-15a05b99b434-audit-log\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.199592 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.199566    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0de6fb0f-5338-4568-a3b6-15a05b99b434-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.200297 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.200266    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0de6fb0f-5338-4568-a3b6-15a05b99b434-metrics-server-audit-profiles\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.201870 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.201791    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0de6fb0f-5338-4568-a3b6-15a05b99b434-client-ca-bundle\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.201975 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.201912    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0de6fb0f-5338-4568-a3b6-15a05b99b434-secret-metrics-server-tls\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.202052 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.202033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0de6fb0f-5338-4568-a3b6-15a05b99b434-secret-metrics-server-client-certs\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.207223 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.207200    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnzph\" (UniqueName: \"kubernetes.io/projected/0de6fb0f-5338-4568-a3b6-15a05b99b434-kube-api-access-qnzph\") pod \"metrics-server-75c58d4475-nwtmf\" (UID: \"0de6fb0f-5338-4568-a3b6-15a05b99b434\") " pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.300443 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.300369    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:18:52.360582 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.360547    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-44mvp"]
Jul 30 00:18:52.363334 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.363308    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-44mvp"
Jul 30 00:18:52.366648 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.366197    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 30 00:18:52.366648 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.366493    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-l4mq6\""
Jul 30 00:18:52.371769 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.371616    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-44mvp"]
Jul 30 00:18:52.401019 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.400982    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9e278fb-935e-4397-8856-630640105426-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-44mvp\" (UID: \"c9e278fb-935e-4397-8856-630640105426\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-44mvp"
Jul 30 00:18:52.401168 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.401040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvxhd\" (UniqueName: \"kubernetes.io/projected/c9e278fb-935e-4397-8856-630640105426-kube-api-access-hvxhd\") pod \"monitoring-plugin-68b7fc9574-44mvp\" (UID: \"c9e278fb-935e-4397-8856-630640105426\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-44mvp"
Jul 30 00:18:52.502384 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.502346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9e278fb-935e-4397-8856-630640105426-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-44mvp\" (UID: \"c9e278fb-935e-4397-8856-630640105426\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-44mvp"
Jul 30 00:18:52.502577 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.502405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvxhd\" (UniqueName: \"kubernetes.io/projected/c9e278fb-935e-4397-8856-630640105426-kube-api-access-hvxhd\") pod \"monitoring-plugin-68b7fc9574-44mvp\" (UID: \"c9e278fb-935e-4397-8856-630640105426\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-44mvp"
Jul 30 00:18:52.502577 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:52.502548    2579 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 30 00:18:52.502658 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:18:52.502644    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e278fb-935e-4397-8856-630640105426-monitoring-plugin-cert podName:c9e278fb-935e-4397-8856-630640105426 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:53.002622113 +0000 UTC m=+58.299889030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/c9e278fb-935e-4397-8856-630640105426-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-44mvp" (UID: "c9e278fb-935e-4397-8856-630640105426") : secret "monitoring-plugin-cert" not found
Jul 30 00:18:52.517820 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.517798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvxhd\" (UniqueName: \"kubernetes.io/projected/c9e278fb-935e-4397-8856-630640105426-kube-api-access-hvxhd\") pod \"monitoring-plugin-68b7fc9574-44mvp\" (UID: \"c9e278fb-935e-4397-8856-630640105426\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-44mvp"
Jul 30 00:18:52.801918 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:52.801887    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-75c58d4475-nwtmf"]
Jul 30 00:18:52.806403 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:18:52.806324    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de6fb0f_5338_4568_a3b6_15a05b99b434.slice/crio-6ff93b396d69c5099c5910cbb8538f98e7a64c616c521687b774bff801cab229 WatchSource:0}: Error finding container 6ff93b396d69c5099c5910cbb8538f98e7a64c616c521687b774bff801cab229: Status 404 returned error can't find the container with id 6ff93b396d69c5099c5910cbb8538f98e7a64c616c521687b774bff801cab229
Jul 30 00:18:53.007688 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.007608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9e278fb-935e-4397-8856-630640105426-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-44mvp\" (UID: \"c9e278fb-935e-4397-8856-630640105426\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-44mvp"
Jul 30 00:18:53.010362 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.010340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9e278fb-935e-4397-8856-630640105426-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-44mvp\" (UID: \"c9e278fb-935e-4397-8856-630640105426\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-44mvp"
Jul 30 00:18:53.279456 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.279371    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-44mvp"
Jul 30 00:18:53.557871 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.557841    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-597b49ddf8-6blvn"]
Jul 30 00:18:53.588998 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.588970    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-44mvp"]
Jul 30 00:18:53.592647 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:18:53.592615    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e278fb_935e_4397_8856_630640105426.slice/crio-cfabd6b849f2bc6a6b7a6a4869d0d97b15fc0b0e422fc894601b31e3d1ef6bd2 WatchSource:0}: Error finding container cfabd6b849f2bc6a6b7a6a4869d0d97b15fc0b0e422fc894601b31e3d1ef6bd2: Status 404 returned error can't find the container with id cfabd6b849f2bc6a6b7a6a4869d0d97b15fc0b0e422fc894601b31e3d1ef6bd2
Jul 30 00:18:53.635149 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.635036    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52138b7c-2f87-486d-855c-dc6bb4c8c39b","Type":"ContainerStarted","Data":"400be706d0bad2580b8b98036a1156cbf3b3a542372701d692045235ec922061"}
Jul 30 00:18:53.638286 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.637887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb" event={"ID":"3aa3fcdd-0019-4f8c-a799-6509d727d472","Type":"ContainerStarted","Data":"91fbf1a36d10a34999a97eea3a6ef69bd5ffaa78a255672a028c411dab2f7eb2"}
Jul 30 00:18:53.638286 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.637954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb" event={"ID":"3aa3fcdd-0019-4f8c-a799-6509d727d472","Type":"ContainerStarted","Data":"d738e91a589198702a7995e034af306f3643acaf1fd98cd0f1f2bed6b59d84d8"}
Jul 30 00:18:53.638286 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.637990    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb" event={"ID":"3aa3fcdd-0019-4f8c-a799-6509d727d472","Type":"ContainerStarted","Data":"d38178886afdd34df67e8130780256accc25dcab765869418f74bae715ea30ec"}
Jul 30 00:18:53.639800 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.639443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-44mvp" event={"ID":"c9e278fb-935e-4397-8856-630640105426","Type":"ContainerStarted","Data":"cfabd6b849f2bc6a6b7a6a4869d0d97b15fc0b0e422fc894601b31e3d1ef6bd2"}
Jul 30 00:18:53.642513 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.642488    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf" event={"ID":"0de6fb0f-5338-4568-a3b6-15a05b99b434","Type":"ContainerStarted","Data":"6ff93b396d69c5099c5910cbb8538f98e7a64c616c521687b774bff801cab229"}
Jul 30 00:18:53.826474 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.826139    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 00:18:53.837319 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.837297    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.842786 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.842727    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 30 00:18:53.842995 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.842961    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 30 00:18:53.843075 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.843021    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rfzmp\""
Jul 30 00:18:53.843680 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.843185    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 30 00:18:53.843680 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.843260    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3vugdlve7uhrl\""
Jul 30 00:18:53.843680 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.843476    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 30 00:18:53.843680 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.843509    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 30 00:18:53.843680 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.843660    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 30 00:18:53.843950 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.843877    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 30 00:18:53.844088 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.844064    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 30 00:18:53.844231 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.844213    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 30 00:18:53.845300 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.844530    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 30 00:18:53.846138 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.845455    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 30 00:18:53.846138 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.845623    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 30 00:18:53.847938 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.847850    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 00:18:53.848830 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.848709    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 30 00:18:53.853558 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.853335    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 30 00:18:53.918420 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.918609 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.918609 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c89a159-e52a-4274-b236-131ecb0128e8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.918609 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.918609 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.918829 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.918829 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c89a159-e52a-4274-b236-131ecb0128e8-config-out\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.918829 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918665    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.918829 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.918829 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-web-config\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.918829 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918738    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w8ml\" (UniqueName: \"kubernetes.io/projected/8c89a159-e52a-4274-b236-131ecb0128e8-kube-api-access-5w8ml\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.918829 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918771    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.918829 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918789    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.918829 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.918829 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.918829 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.919490 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-config\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.919490 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.919490 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:53.919490 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:53.918929    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.019613 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.019577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.019613 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.019617    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.019835 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.019663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.019835 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.019689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.019835 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.019717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.019835 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.019743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c89a159-e52a-4274-b236-131ecb0128e8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.019835 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.019772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.019835 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.019833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.020034 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.019891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.020034 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.019923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c89a159-e52a-4274-b236-131ecb0128e8-config-out\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.020034 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.019962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.020034 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.019987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.020163 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.020032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-web-config\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.020163 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.020062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w8ml\" (UniqueName: \"kubernetes.io/projected/8c89a159-e52a-4274-b236-131ecb0128e8-kube-api-access-5w8ml\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.020163 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.020114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.020163 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.020141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.020304 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.020190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.020304 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.020216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.020304 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.020267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.020402 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.020305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-config\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.020402 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.020386    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.020894 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.020867    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.021012 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.020929    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.023141 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.023071    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.023620 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.023483    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.023921 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.023877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.025018 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.024971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.026059 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.026010    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c89a159-e52a-4274-b236-131ecb0128e8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.026266 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.026208    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.026903 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.026796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.027111 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.027048    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.027800 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.027749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.028430 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.028386    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-web-config\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.030371 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.028988    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.030371 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.029586    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w8ml\" (UniqueName: \"kubernetes.io/projected/8c89a159-e52a-4274-b236-131ecb0128e8-kube-api-access-5w8ml\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.030371 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.030159    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c89a159-e52a-4274-b236-131ecb0128e8-config-out\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.030371 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.030315    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-config\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.030610 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.030453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.030847 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.030823    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.031419 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.031401    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.157909 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.157870    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:18:54.341092 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.341061    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 00:18:54.399336 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:18:54.399285    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c89a159_e52a_4274_b236_131ecb0128e8.slice/crio-8d982dd04ac4797639cc6a48e9f471fbe8c63c974ef1a0a674532cca9c2a8c6b WatchSource:0}: Error finding container 8d982dd04ac4797639cc6a48e9f471fbe8c63c974ef1a0a674532cca9c2a8c6b: Status 404 returned error can't find the container with id 8d982dd04ac4797639cc6a48e9f471fbe8c63c974ef1a0a674532cca9c2a8c6b
Jul 30 00:18:54.649388 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.649275    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c89a159-e52a-4274-b236-131ecb0128e8","Type":"ContainerStarted","Data":"8d982dd04ac4797639cc6a48e9f471fbe8c63c974ef1a0a674532cca9c2a8c6b"}
Jul 30 00:18:54.653813 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.653743    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52138b7c-2f87-486d-855c-dc6bb4c8c39b","Type":"ContainerStarted","Data":"c68bbf17bce8523d10159185f16bc694efd668aef8f046069016ad4102c73a66"}
Jul 30 00:18:54.653974 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.653825    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52138b7c-2f87-486d-855c-dc6bb4c8c39b","Type":"ContainerStarted","Data":"560ff3c9b7441b375cf15a4b04579cbae82d1db516d2fed8da6062b4e5e06610"}
Jul 30 00:18:54.653974 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.653867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52138b7c-2f87-486d-855c-dc6bb4c8c39b","Type":"ContainerStarted","Data":"50d276187536202cd8838a2115e87582861db8cb14a58c4d021ae7c861bd9467"}
Jul 30 00:18:54.653974 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:54.653902    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52138b7c-2f87-486d-855c-dc6bb4c8c39b","Type":"ContainerStarted","Data":"e6917acfa96fe1966c6e0c95ce953cc1afa534b54f15490993b77f4a80a6ac8b"}
Jul 30 00:18:55.495860 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:55.495823    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-slhpw"
Jul 30 00:18:55.658859 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:55.658831    2579 generic.go:356] "Generic (PLEG): container finished" podID="8c89a159-e52a-4274-b236-131ecb0128e8" containerID="777b3ac55e59c8a495a8c68c36208b1426029f1df494b171e1e807ae1d2f105a" exitCode=0
Jul 30 00:18:55.658986 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:55.658885    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c89a159-e52a-4274-b236-131ecb0128e8","Type":"ContainerDied","Data":"777b3ac55e59c8a495a8c68c36208b1426029f1df494b171e1e807ae1d2f105a"}
Jul 30 00:18:55.662651 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:55.662619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52138b7c-2f87-486d-855c-dc6bb4c8c39b","Type":"ContainerStarted","Data":"1478560230ebfce046448e3e4e3dda3e275f8902ac84f95d1509c0071ad8c55e"}
Jul 30 00:18:55.666049 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:55.666022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb" event={"ID":"3aa3fcdd-0019-4f8c-a799-6509d727d472","Type":"ContainerStarted","Data":"c3466cdf82cb7c7280d4ef3f0f70e8c0faee9c944c1ac7396631850fd774f7c7"}
Jul 30 00:18:55.666153 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:55.666091    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:18:55.666153 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:55.666124    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb" event={"ID":"3aa3fcdd-0019-4f8c-a799-6509d727d472","Type":"ContainerStarted","Data":"9b2d5740c11311177d530baf6f066f450b0278c36cf290ddd1c1e785c608795d"}
Jul 30 00:18:55.666153 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:55.666149    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb" event={"ID":"3aa3fcdd-0019-4f8c-a799-6509d727d472","Type":"ContainerStarted","Data":"7a464e568651c20b6766085a3d5afcf45dc66e144fce423f838eb2da1122c9ed"}
Jul 30 00:18:55.667444 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:55.667422    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-44mvp" event={"ID":"c9e278fb-935e-4397-8856-630640105426","Type":"ContainerStarted","Data":"8651cd63ff0cb1a25bfdae666d3f7a2b386c46af5dca3ba4f96bee774290aea7"}
Jul 30 00:18:55.667568 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:55.667553    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-44mvp"
Jul 30 00:18:55.668958 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:55.668936    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf" event={"ID":"0de6fb0f-5338-4568-a3b6-15a05b99b434","Type":"ContainerStarted","Data":"928b816b74f7dc4d3eb34f07681187b68597fdcf9e5afd7c78fe8fe69374b178"}
Jul 30 00:18:55.672657 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:55.672637    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-44mvp"
Jul 30 00:18:55.701970 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:55.701922    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-44mvp" podStartSLOduration=1.7282559119999998 podStartE2EDuration="3.701908937s" totalImagesPullingTime="1.973653025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:55.700815333 +0000 UTC m=+60.998082265" watchObservedRunningTime="2026-07-30 00:18:55.701908937 +0000 UTC m=+60.999175847"
Jul 30 00:18:55.729291 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:55.729216    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.483299177 podStartE2EDuration="7.729196614s" totalImagesPullingTime="4.228972615s" totalInitContainerRuntime="16.924822ms" isStatefulPod=false podCreationTimestamp="2026-07-30 00:18:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:55.727299242 +0000 UTC m=+61.024566176" watchObservedRunningTime="2026-07-30 00:18:55.729196614 +0000 UTC m=+61.026463547"
Jul 30 00:18:55.749907 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:55.749819    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb" podStartSLOduration=2.281756775 podStartE2EDuration="6.7498024s" totalImagesPullingTime="4.468045625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:18:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:55.748631 +0000 UTC m=+61.045897932" watchObservedRunningTime="2026-07-30 00:18:55.7498024 +0000 UTC m=+61.047069332"
Jul 30 00:18:55.765370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:55.765325    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf" podStartSLOduration=2.725624948 podStartE2EDuration="4.765297193s" totalImagesPullingTime="2.039672245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:55.764650835 +0000 UTC m=+61.061917767" watchObservedRunningTime="2026-07-30 00:18:55.765297193 +0000 UTC m=+61.062564124"
Jul 30 00:18:59.693934 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:59.693891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c89a159-e52a-4274-b236-131ecb0128e8","Type":"ContainerStarted","Data":"dc32b0fc6601fdb6c158e50a53384261c3334593c0b82ae31942e81bbdca22d6"}
Jul 30 00:18:59.694382 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:59.693979    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c89a159-e52a-4274-b236-131ecb0128e8","Type":"ContainerStarted","Data":"8a67756d1e075940c3bf5f8a3f0c3d1a8819fa26975900e1214f2fb6c4783346"}
Jul 30 00:18:59.694382 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:59.694020    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c89a159-e52a-4274-b236-131ecb0128e8","Type":"ContainerStarted","Data":"420e56456e90504ae62fe3f9aa0a127028363554fd3e12743ade7c7f5350feeb"}
Jul 30 00:18:59.694382 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:59.694060    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c89a159-e52a-4274-b236-131ecb0128e8","Type":"ContainerStarted","Data":"3f439dd127c49f058eda08eb41eab0759c1a0df4bd8b1d975d52c819f0d00e17"}
Jul 30 00:18:59.694382 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:59.694103    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c89a159-e52a-4274-b236-131ecb0128e8","Type":"ContainerStarted","Data":"0162c846c23b5cad1f94fac39edc2cf93ca2d87a48829ea5c51bf17a8e5c1f72"}
Jul 30 00:18:59.694382 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:59.694139    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c89a159-e52a-4274-b236-131ecb0128e8","Type":"ContainerStarted","Data":"e5454c4550a85b91a8c6588292e35f8f7b21590debf8cd49750b5ddbe9072477"}
Jul 30 00:18:59.727327 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:18:59.727272    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.400189573 podStartE2EDuration="6.727238711s" totalImagesPullingTime="3.302395985s" totalInitContainerRuntime="24.653153ms" isStatefulPod=false podCreationTimestamp="2026-07-30 00:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:59.725055036 +0000 UTC m=+65.022321982" watchObservedRunningTime="2026-07-30 00:18:59.727238711 +0000 UTC m=+65.024505691"
Jul 30 00:19:00.893813 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:00.893776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs\") pod \"network-metrics-daemon-b6ns2\" (UID: \"a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1\") " pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:19:00.894212 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:00.893880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret\") pod \"global-pull-secret-syncer-86f4q\" (UID: \"ebd8b209-a805-4976-bb8c-24cba1f01d86\") " pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:19:00.896878 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:00.896854    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 30 00:19:00.897988 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:00.897971    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 30 00:19:00.906574 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:00.906539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebd8b209-a805-4976-bb8c-24cba1f01d86-original-pull-secret\") pod \"global-pull-secret-syncer-86f4q\" (UID: \"ebd8b209-a805-4976-bb8c-24cba1f01d86\") " pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:19:00.906696 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:00.906552    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1-metrics-certs\") pod \"network-metrics-daemon-b6ns2\" (UID: \"a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1\") " pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:19:00.930275 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:00.930236    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f7pzr\""
Jul 30 00:19:00.937694 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:00.937676    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6ns2"
Jul 30 00:19:01.035287 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:01.035225    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86f4q"
Jul 30 00:19:01.095107 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:01.095068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmm96\" (UniqueName: \"kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96\") pod \"network-check-target-hg9rq\" (UID: \"3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0\") " pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:19:01.098497 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:01.098467    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 30 00:19:01.099518 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:01.099497    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b6ns2"]
Jul 30 00:19:01.102570 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:19:01.102537    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3bb43f3_fe55_4cf2_8ee3_322ea5d467b1.slice/crio-3861e54b23f055a2e4dea7ccdc1ed45f1cc7d96b8c6f774fbd2df5962c2fbb16 WatchSource:0}: Error finding container 3861e54b23f055a2e4dea7ccdc1ed45f1cc7d96b8c6f774fbd2df5962c2fbb16: Status 404 returned error can't find the container with id 3861e54b23f055a2e4dea7ccdc1ed45f1cc7d96b8c6f774fbd2df5962c2fbb16
Jul 30 00:19:01.108519 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:01.108454    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 30 00:19:01.120161 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:01.120135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmm96\" (UniqueName: \"kubernetes.io/projected/3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0-kube-api-access-bmm96\") pod \"network-check-target-hg9rq\" (UID: \"3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0\") " pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:19:01.185024 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:01.184946    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-86f4q"]
Jul 30 00:19:01.187898 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:19:01.187870    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd8b209_a805_4976_bb8c_24cba1f01d86.slice/crio-0961113b31f6bb61dafe8f56ca06aa6d3fb6abf8b6c7420336903b00960711ba WatchSource:0}: Error finding container 0961113b31f6bb61dafe8f56ca06aa6d3fb6abf8b6c7420336903b00960711ba: Status 404 returned error can't find the container with id 0961113b31f6bb61dafe8f56ca06aa6d3fb6abf8b6c7420336903b00960711ba
Jul 30 00:19:01.234823 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:01.234799    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qvw7x\""
Jul 30 00:19:01.243152 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:01.243119    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:19:01.391659 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:01.391527    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hg9rq"]
Jul 30 00:19:01.394203 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:19:01.394177    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd5fe7f_e640_40ce_b2db_a5ef684bc5f0.slice/crio-9d865f05dab53c8ec0441fb80c5faae02fcfd662436d3f3dfb21485d9f34004a WatchSource:0}: Error finding container 9d865f05dab53c8ec0441fb80c5faae02fcfd662436d3f3dfb21485d9f34004a: Status 404 returned error can't find the container with id 9d865f05dab53c8ec0441fb80c5faae02fcfd662436d3f3dfb21485d9f34004a
Jul 30 00:19:01.682739 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:01.682663    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b799ccf8b-62djb"
Jul 30 00:19:01.711069 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:01.711029    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b6ns2" event={"ID":"a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1","Type":"ContainerStarted","Data":"3861e54b23f055a2e4dea7ccdc1ed45f1cc7d96b8c6f774fbd2df5962c2fbb16"}
Jul 30 00:19:01.713473 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:01.713438    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-86f4q" event={"ID":"ebd8b209-a805-4976-bb8c-24cba1f01d86","Type":"ContainerStarted","Data":"0961113b31f6bb61dafe8f56ca06aa6d3fb6abf8b6c7420336903b00960711ba"}
Jul 30 00:19:01.716977 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:01.716945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hg9rq" event={"ID":"3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0","Type":"ContainerStarted","Data":"9d865f05dab53c8ec0441fb80c5faae02fcfd662436d3f3dfb21485d9f34004a"}
Jul 30 00:19:02.728050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:02.728004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b6ns2" event={"ID":"a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1","Type":"ContainerStarted","Data":"9d19ca19b2f0a4cc78ab96e70fa79b286e39246db2c91de240fc57a0cc13cda9"}
Jul 30 00:19:02.728551 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:02.728085    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b6ns2" event={"ID":"a3bb43f3-fe55-4cf2-8ee3-322ea5d467b1","Type":"ContainerStarted","Data":"4011710eea4139140e4c9e367236faa88f5985da178bcd32365112b2cb7faf86"}
Jul 30 00:19:02.749932 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:02.749145    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-b6ns2" podStartSLOduration=66.71731474 podStartE2EDuration="1m7.749124557s" totalImagesPullingTime="1.031809817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:19:02.748418646 +0000 UTC m=+68.045685579" watchObservedRunningTime="2026-07-30 00:19:02.749124557 +0000 UTC m=+68.046391489"
Jul 30 00:19:04.158955 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:04.158917    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:06.748580 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:06.748535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hg9rq" event={"ID":"3cd5fe7f-e640-40ce-b2db-a5ef684bc5f0","Type":"ContainerStarted","Data":"090e061a3413839c4a4bc664e22d17980510104b8ed4c76b798278f2c845fbce"}
Jul 30 00:19:06.749123 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:06.748606    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:19:06.750942 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:06.750910    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-86f4q" event={"ID":"ebd8b209-a805-4976-bb8c-24cba1f01d86","Type":"ContainerStarted","Data":"fca3ae483e8582bc780b11712edcc265b7fb32ae9f6db4305b26b9facac34522"}
Jul 30 00:19:06.764651 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:06.764604    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hg9rq" podStartSLOduration=67.194299374 podStartE2EDuration="1m11.764587449s" totalImagesPullingTime="4.570288075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:19:06.763438189 +0000 UTC m=+72.060705120" watchObservedRunningTime="2026-07-30 00:19:06.764587449 +0000 UTC m=+72.061854372"
Jul 30 00:19:06.778544 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:06.778499    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-86f4q" podStartSLOduration=66.993755776 podStartE2EDuration="1m11.778480527s" totalImagesPullingTime="4.784724751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:19:06.777541469 +0000 UTC m=+72.074808401" watchObservedRunningTime="2026-07-30 00:19:06.778480527 +0000 UTC m=+72.075747458"
Jul 30 00:19:10.691861 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:10.691828    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:10.756746 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:10.756718    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:10.781770 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:10.781746    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:12.301095 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:12.301057    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:19:12.301518 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:12.301133    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:19:18.584516 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.584348    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-597b49ddf8-6blvn" podUID="c855f937-8419-4e43-ab83-46aae0c940b5" containerName="console" containerID="cri-o://cdfe8ecb2ca7a51cc5e8e1af307db18641c10dcc1f67601c5e1a5af4d937c2a9" gracePeriod=15
Jul 30 00:19:18.800015 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.799983    2579 generic.go:356] "Generic (PLEG): container finished" podID="c855f937-8419-4e43-ab83-46aae0c940b5" containerID="cdfe8ecb2ca7a51cc5e8e1af307db18641c10dcc1f67601c5e1a5af4d937c2a9" exitCode=0
Jul 30 00:19:18.800181 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.800062    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597b49ddf8-6blvn" event={"ID":"c855f937-8419-4e43-ab83-46aae0c940b5","Type":"ContainerDied","Data":"cdfe8ecb2ca7a51cc5e8e1af307db18641c10dcc1f67601c5e1a5af4d937c2a9"}
Jul 30 00:19:18.832718 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.832695    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:19:18.852174 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.852100    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c855f937-8419-4e43-ab83-46aae0c940b5-console-oauth-config\") pod \"c855f937-8419-4e43-ab83-46aae0c940b5\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") "
Jul 30 00:19:18.852174 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.852162    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c855f937-8419-4e43-ab83-46aae0c940b5-service-ca\") pod \"c855f937-8419-4e43-ab83-46aae0c940b5\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") "
Jul 30 00:19:18.852384 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.852332    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qqs9\" (UniqueName: \"kubernetes.io/projected/c855f937-8419-4e43-ab83-46aae0c940b5-kube-api-access-8qqs9\") pod \"c855f937-8419-4e43-ab83-46aae0c940b5\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") "
Jul 30 00:19:18.852384 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.852369    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c855f937-8419-4e43-ab83-46aae0c940b5-oauth-serving-cert\") pod \"c855f937-8419-4e43-ab83-46aae0c940b5\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") "
Jul 30 00:19:18.852490 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.852420    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c855f937-8419-4e43-ab83-46aae0c940b5-console-config\") pod \"c855f937-8419-4e43-ab83-46aae0c940b5\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") "
Jul 30 00:19:18.852490 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.852446    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c855f937-8419-4e43-ab83-46aae0c940b5-console-serving-cert\") pod \"c855f937-8419-4e43-ab83-46aae0c940b5\" (UID: \"c855f937-8419-4e43-ab83-46aae0c940b5\") "
Jul 30 00:19:18.852591 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.852538    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c855f937-8419-4e43-ab83-46aae0c940b5-service-ca" (OuterVolumeSpecName: "service-ca") pod "c855f937-8419-4e43-ab83-46aae0c940b5" (UID: "c855f937-8419-4e43-ab83-46aae0c940b5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:19:18.852779 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.852740    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c855f937-8419-4e43-ab83-46aae0c940b5-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:18.852919 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.852836    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c855f937-8419-4e43-ab83-46aae0c940b5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c855f937-8419-4e43-ab83-46aae0c940b5" (UID: "c855f937-8419-4e43-ab83-46aae0c940b5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:19:18.852919 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.852888    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c855f937-8419-4e43-ab83-46aae0c940b5-console-config" (OuterVolumeSpecName: "console-config") pod "c855f937-8419-4e43-ab83-46aae0c940b5" (UID: "c855f937-8419-4e43-ab83-46aae0c940b5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:19:18.855386 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.855089    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c855f937-8419-4e43-ab83-46aae0c940b5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c855f937-8419-4e43-ab83-46aae0c940b5" (UID: "c855f937-8419-4e43-ab83-46aae0c940b5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:19:18.855386 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.855282    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c855f937-8419-4e43-ab83-46aae0c940b5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c855f937-8419-4e43-ab83-46aae0c940b5" (UID: "c855f937-8419-4e43-ab83-46aae0c940b5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:19:18.855545 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.855453    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c855f937-8419-4e43-ab83-46aae0c940b5-kube-api-access-8qqs9" (OuterVolumeSpecName: "kube-api-access-8qqs9") pod "c855f937-8419-4e43-ab83-46aae0c940b5" (UID: "c855f937-8419-4e43-ab83-46aae0c940b5"). InnerVolumeSpecName "kube-api-access-8qqs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:19:18.953335 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.953301    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c855f937-8419-4e43-ab83-46aae0c940b5-console-oauth-config\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:18.953335 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.953328    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qqs9\" (UniqueName: \"kubernetes.io/projected/c855f937-8419-4e43-ab83-46aae0c940b5-kube-api-access-8qqs9\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:18.953335 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.953341    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c855f937-8419-4e43-ab83-46aae0c940b5-oauth-serving-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:18.953596 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.953356    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c855f937-8419-4e43-ab83-46aae0c940b5-console-config\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:18.953596 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:18.953369    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c855f937-8419-4e43-ab83-46aae0c940b5-console-serving-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:19.805354 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:19.805315    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597b49ddf8-6blvn" event={"ID":"c855f937-8419-4e43-ab83-46aae0c940b5","Type":"ContainerDied","Data":"7be5b007aaad025dd9ce9b1a980da2a91410c313746219a95c85eb93ae80de44"}
Jul 30 00:19:19.805354 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:19.805350    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597b49ddf8-6blvn"
Jul 30 00:19:19.805850 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:19.805415    2579 scope.go:117] "RemoveContainer" containerID="cdfe8ecb2ca7a51cc5e8e1af307db18641c10dcc1f67601c5e1a5af4d937c2a9"
Jul 30 00:19:19.824045 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:19.824017    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-597b49ddf8-6blvn"]
Jul 30 00:19:19.827576 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:19.827553    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-597b49ddf8-6blvn"]
Jul 30 00:19:21.322166 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:21.322129    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c855f937-8419-4e43-ab83-46aae0c940b5" path="/var/lib/kubelet/pods/c855f937-8419-4e43-ab83-46aae0c940b5/volumes"
Jul 30 00:19:32.305516 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:32.305485    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:19:32.309317 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:32.309294    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-75c58d4475-nwtmf"
Jul 30 00:19:37.322221 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:37.322191    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 00:19:37.322658 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:37.322496    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="alertmanager" containerID="cri-o://400be706d0bad2580b8b98036a1156cbf3b3a542372701d692045235ec922061" gracePeriod=120
Jul 30 00:19:37.322658 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:37.322548    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="kube-rbac-proxy-metric" containerID="cri-o://560ff3c9b7441b375cf15a4b04579cbae82d1db516d2fed8da6062b4e5e06610" gracePeriod=120
Jul 30 00:19:37.322658 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:37.322582    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="config-reloader" containerID="cri-o://c68bbf17bce8523d10159185f16bc694efd668aef8f046069016ad4102c73a66" gracePeriod=120
Jul 30 00:19:37.322658 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:37.322569    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="kube-rbac-proxy-web" containerID="cri-o://e6917acfa96fe1966c6e0c95ce953cc1afa534b54f15490993b77f4a80a6ac8b" gracePeriod=120
Jul 30 00:19:37.322859 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:37.322582    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="kube-rbac-proxy" containerID="cri-o://50d276187536202cd8838a2115e87582861db8cb14a58c4d021ae7c861bd9467" gracePeriod=120
Jul 30 00:19:37.322859 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:37.322662    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="prom-label-proxy" containerID="cri-o://1478560230ebfce046448e3e4e3dda3e275f8902ac84f95d1509c0071ad8c55e" gracePeriod=120
Jul 30 00:19:37.757659 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:37.757628    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hg9rq"
Jul 30 00:19:37.885167 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:37.885134    2579 generic.go:356] "Generic (PLEG): container finished" podID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerID="1478560230ebfce046448e3e4e3dda3e275f8902ac84f95d1509c0071ad8c55e" exitCode=0
Jul 30 00:19:37.885167 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:37.885161    2579 generic.go:356] "Generic (PLEG): container finished" podID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerID="50d276187536202cd8838a2115e87582861db8cb14a58c4d021ae7c861bd9467" exitCode=0
Jul 30 00:19:37.885167 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:37.885169    2579 generic.go:356] "Generic (PLEG): container finished" podID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerID="c68bbf17bce8523d10159185f16bc694efd668aef8f046069016ad4102c73a66" exitCode=0
Jul 30 00:19:37.885167 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:37.885175    2579 generic.go:356] "Generic (PLEG): container finished" podID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerID="400be706d0bad2580b8b98036a1156cbf3b3a542372701d692045235ec922061" exitCode=0
Jul 30 00:19:37.885440 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:37.885207    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52138b7c-2f87-486d-855c-dc6bb4c8c39b","Type":"ContainerDied","Data":"1478560230ebfce046448e3e4e3dda3e275f8902ac84f95d1509c0071ad8c55e"}
Jul 30 00:19:37.885440 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:37.885320    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52138b7c-2f87-486d-855c-dc6bb4c8c39b","Type":"ContainerDied","Data":"50d276187536202cd8838a2115e87582861db8cb14a58c4d021ae7c861bd9467"}
Jul 30 00:19:37.885440 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:37.885353    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52138b7c-2f87-486d-855c-dc6bb4c8c39b","Type":"ContainerDied","Data":"c68bbf17bce8523d10159185f16bc694efd668aef8f046069016ad4102c73a66"}
Jul 30 00:19:37.885440 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:37.885379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52138b7c-2f87-486d-855c-dc6bb4c8c39b","Type":"ContainerDied","Data":"400be706d0bad2580b8b98036a1156cbf3b3a542372701d692045235ec922061"}
Jul 30 00:19:38.600015 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.599991    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:38.711289 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.711189    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52138b7c-2f87-486d-855c-dc6bb4c8c39b-alertmanager-trusted-ca-bundle\") pod \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") "
Jul 30 00:19:38.711289 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.711230    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-kube-rbac-proxy\") pod \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") "
Jul 30 00:19:38.711289 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.711267    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-cluster-tls-config\") pod \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") "
Jul 30 00:19:38.711289 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.711291    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/52138b7c-2f87-486d-855c-dc6bb4c8c39b-config-out\") pod \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") "
Jul 30 00:19:38.711618 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.711447    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-config-volume\") pod \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") "
Jul 30 00:19:38.711618 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.711494    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7fbj\" (UniqueName: \"kubernetes.io/projected/52138b7c-2f87-486d-855c-dc6bb4c8c39b-kube-api-access-m7fbj\") pod \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") "
Jul 30 00:19:38.711618 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.711521    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-main-tls\") pod \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") "
Jul 30 00:19:38.711618 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.711552    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/52138b7c-2f87-486d-855c-dc6bb4c8c39b-tls-assets\") pod \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") "
Jul 30 00:19:38.711618 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.711610    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52138b7c-2f87-486d-855c-dc6bb4c8c39b-metrics-client-ca\") pod \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") "
Jul 30 00:19:38.711873 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.711678    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52138b7c-2f87-486d-855c-dc6bb4c8c39b-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "52138b7c-2f87-486d-855c-dc6bb4c8c39b" (UID: "52138b7c-2f87-486d-855c-dc6bb4c8c39b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:19:38.711873 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.711703    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") "
Jul 30 00:19:38.711873 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.711734    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-kube-rbac-proxy-web\") pod \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") "
Jul 30 00:19:38.711873 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.711768    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/52138b7c-2f87-486d-855c-dc6bb4c8c39b-alertmanager-main-db\") pod \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") "
Jul 30 00:19:38.711873 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.711798    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-web-config\") pod \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\" (UID: \"52138b7c-2f87-486d-855c-dc6bb4c8c39b\") "
Jul 30 00:19:38.712131 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.712077    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52138b7c-2f87-486d-855c-dc6bb4c8c39b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:38.712677 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.712616    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52138b7c-2f87-486d-855c-dc6bb4c8c39b-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "52138b7c-2f87-486d-855c-dc6bb4c8c39b" (UID: "52138b7c-2f87-486d-855c-dc6bb4c8c39b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:19:38.713087 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.713058    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52138b7c-2f87-486d-855c-dc6bb4c8c39b-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "52138b7c-2f87-486d-855c-dc6bb4c8c39b" (UID: "52138b7c-2f87-486d-855c-dc6bb4c8c39b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:19:38.714617 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.714486    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-config-volume" (OuterVolumeSpecName: "config-volume") pod "52138b7c-2f87-486d-855c-dc6bb4c8c39b" (UID: "52138b7c-2f87-486d-855c-dc6bb4c8c39b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:19:38.714881 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.714848    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52138b7c-2f87-486d-855c-dc6bb4c8c39b-config-out" (OuterVolumeSpecName: "config-out") pod "52138b7c-2f87-486d-855c-dc6bb4c8c39b" (UID: "52138b7c-2f87-486d-855c-dc6bb4c8c39b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:19:38.715083 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.715040    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52138b7c-2f87-486d-855c-dc6bb4c8c39b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "52138b7c-2f87-486d-855c-dc6bb4c8c39b" (UID: "52138b7c-2f87-486d-855c-dc6bb4c8c39b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:19:38.715508 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.715472    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "52138b7c-2f87-486d-855c-dc6bb4c8c39b" (UID: "52138b7c-2f87-486d-855c-dc6bb4c8c39b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:19:38.715672 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.715641    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "52138b7c-2f87-486d-855c-dc6bb4c8c39b" (UID: "52138b7c-2f87-486d-855c-dc6bb4c8c39b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:19:38.715874 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.715850    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52138b7c-2f87-486d-855c-dc6bb4c8c39b-kube-api-access-m7fbj" (OuterVolumeSpecName: "kube-api-access-m7fbj") pod "52138b7c-2f87-486d-855c-dc6bb4c8c39b" (UID: "52138b7c-2f87-486d-855c-dc6bb4c8c39b"). InnerVolumeSpecName "kube-api-access-m7fbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:19:38.715978 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.715961    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "52138b7c-2f87-486d-855c-dc6bb4c8c39b" (UID: "52138b7c-2f87-486d-855c-dc6bb4c8c39b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:19:38.716387 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.716369    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "52138b7c-2f87-486d-855c-dc6bb4c8c39b" (UID: "52138b7c-2f87-486d-855c-dc6bb4c8c39b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:19:38.719275 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.719124    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "52138b7c-2f87-486d-855c-dc6bb4c8c39b" (UID: "52138b7c-2f87-486d-855c-dc6bb4c8c39b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:19:38.724978 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.724957    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-web-config" (OuterVolumeSpecName: "web-config") pod "52138b7c-2f87-486d-855c-dc6bb4c8c39b" (UID: "52138b7c-2f87-486d-855c-dc6bb4c8c39b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:19:38.812826 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.812801    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:38.812826 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.812822    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:38.812966 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.812834    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/52138b7c-2f87-486d-855c-dc6bb4c8c39b-alertmanager-main-db\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:38.812966 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.812844    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-web-config\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:38.812966 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.812854    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:38.812966 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.812863    2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-cluster-tls-config\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:38.812966 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.812871    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/52138b7c-2f87-486d-855c-dc6bb4c8c39b-config-out\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:38.812966 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.812879    2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:38.812966 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.812886    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7fbj\" (UniqueName: \"kubernetes.io/projected/52138b7c-2f87-486d-855c-dc6bb4c8c39b-kube-api-access-m7fbj\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:38.812966 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.812894    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/52138b7c-2f87-486d-855c-dc6bb4c8c39b-secret-alertmanager-main-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:38.812966 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.812904    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/52138b7c-2f87-486d-855c-dc6bb4c8c39b-tls-assets\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:38.812966 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.812913    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52138b7c-2f87-486d-855c-dc6bb4c8c39b-metrics-client-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:38.892179 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.892144    2579 generic.go:356] "Generic (PLEG): container finished" podID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerID="560ff3c9b7441b375cf15a4b04579cbae82d1db516d2fed8da6062b4e5e06610" exitCode=0
Jul 30 00:19:38.892179 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.892176    2579 generic.go:356] "Generic (PLEG): container finished" podID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerID="e6917acfa96fe1966c6e0c95ce953cc1afa534b54f15490993b77f4a80a6ac8b" exitCode=0
Jul 30 00:19:38.892374 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.892256    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:38.892374 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.892236    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52138b7c-2f87-486d-855c-dc6bb4c8c39b","Type":"ContainerDied","Data":"560ff3c9b7441b375cf15a4b04579cbae82d1db516d2fed8da6062b4e5e06610"}
Jul 30 00:19:38.892374 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.892351    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52138b7c-2f87-486d-855c-dc6bb4c8c39b","Type":"ContainerDied","Data":"e6917acfa96fe1966c6e0c95ce953cc1afa534b54f15490993b77f4a80a6ac8b"}
Jul 30 00:19:38.892512 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.892401    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52138b7c-2f87-486d-855c-dc6bb4c8c39b","Type":"ContainerDied","Data":"2fea0dd37cb6fc37e387e1a8660b8c9456a0102077de34ef2f7122df2d40025c"}
Jul 30 00:19:38.892512 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.892418    2579 scope.go:117] "RemoveContainer" containerID="1478560230ebfce046448e3e4e3dda3e275f8902ac84f95d1509c0071ad8c55e"
Jul 30 00:19:38.900451 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.900416    2579 scope.go:117] "RemoveContainer" containerID="560ff3c9b7441b375cf15a4b04579cbae82d1db516d2fed8da6062b4e5e06610"
Jul 30 00:19:38.907566 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.907548    2579 scope.go:117] "RemoveContainer" containerID="50d276187536202cd8838a2115e87582861db8cb14a58c4d021ae7c861bd9467"
Jul 30 00:19:38.914640 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.914617    2579 scope.go:117] "RemoveContainer" containerID="e6917acfa96fe1966c6e0c95ce953cc1afa534b54f15490993b77f4a80a6ac8b"
Jul 30 00:19:38.915772 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.915699    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 00:19:38.920515 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.920492    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 00:19:38.922758 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.922744    2579 scope.go:117] "RemoveContainer" containerID="c68bbf17bce8523d10159185f16bc694efd668aef8f046069016ad4102c73a66"
Jul 30 00:19:38.929927 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.929906    2579 scope.go:117] "RemoveContainer" containerID="400be706d0bad2580b8b98036a1156cbf3b3a542372701d692045235ec922061"
Jul 30 00:19:38.937361 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.937346    2579 scope.go:117] "RemoveContainer" containerID="2f140f96642b0c5aa1503411fe451d423c257dfd7ab1e1452506e8bd9f250661"
Jul 30 00:19:38.944491 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.944464    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 00:19:38.944818 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.944734    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="kube-rbac-proxy"
Jul 30 00:19:38.944818 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.944745    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="kube-rbac-proxy"
Jul 30 00:19:38.944818 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.944757    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="alertmanager"
Jul 30 00:19:38.944818 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.944763    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="alertmanager"
Jul 30 00:19:38.944818 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.944778    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="init-config-reloader"
Jul 30 00:19:38.944818 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.944786    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="init-config-reloader"
Jul 30 00:19:38.944818 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.944797    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="kube-rbac-proxy-web"
Jul 30 00:19:38.944818 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.944805    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="kube-rbac-proxy-web"
Jul 30 00:19:38.944818 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.944819    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c855f937-8419-4e43-ab83-46aae0c940b5" containerName="console"
Jul 30 00:19:38.945118 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.944828    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c855f937-8419-4e43-ab83-46aae0c940b5" containerName="console"
Jul 30 00:19:38.945118 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.944838    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="kube-rbac-proxy-metric"
Jul 30 00:19:38.945118 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.944847    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="kube-rbac-proxy-metric"
Jul 30 00:19:38.945118 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.944856    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="prom-label-proxy"
Jul 30 00:19:38.945118 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.944861    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="prom-label-proxy"
Jul 30 00:19:38.945118 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.944863    2579 scope.go:117] "RemoveContainer" containerID="1478560230ebfce046448e3e4e3dda3e275f8902ac84f95d1509c0071ad8c55e"
Jul 30 00:19:38.945118 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.944868    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="config-reloader"
Jul 30 00:19:38.945118 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.944939    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="config-reloader"
Jul 30 00:19:38.945118 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.945038    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="kube-rbac-proxy-web"
Jul 30 00:19:38.945118 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.945051    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="kube-rbac-proxy"
Jul 30 00:19:38.945118 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.945061    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="prom-label-proxy"
Jul 30 00:19:38.945118 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.945070    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c855f937-8419-4e43-ab83-46aae0c940b5" containerName="console"
Jul 30 00:19:38.945118 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.945079    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="config-reloader"
Jul 30 00:19:38.945118 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.945087    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="alertmanager"
Jul 30 00:19:38.945118 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.945095    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" containerName="kube-rbac-proxy-metric"
Jul 30 00:19:38.945604 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:19:38.945173    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1478560230ebfce046448e3e4e3dda3e275f8902ac84f95d1509c0071ad8c55e\": container with ID starting with 1478560230ebfce046448e3e4e3dda3e275f8902ac84f95d1509c0071ad8c55e not found: ID does not exist" containerID="1478560230ebfce046448e3e4e3dda3e275f8902ac84f95d1509c0071ad8c55e"
Jul 30 00:19:38.945604 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.945201    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1478560230ebfce046448e3e4e3dda3e275f8902ac84f95d1509c0071ad8c55e"} err="failed to get container status \"1478560230ebfce046448e3e4e3dda3e275f8902ac84f95d1509c0071ad8c55e\": rpc error: code = NotFound desc = could not find container \"1478560230ebfce046448e3e4e3dda3e275f8902ac84f95d1509c0071ad8c55e\": container with ID starting with 1478560230ebfce046448e3e4e3dda3e275f8902ac84f95d1509c0071ad8c55e not found: ID does not exist"
Jul 30 00:19:38.945604 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.945264    2579 scope.go:117] "RemoveContainer" containerID="560ff3c9b7441b375cf15a4b04579cbae82d1db516d2fed8da6062b4e5e06610"
Jul 30 00:19:38.945604 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:19:38.945513    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560ff3c9b7441b375cf15a4b04579cbae82d1db516d2fed8da6062b4e5e06610\": container with ID starting with 560ff3c9b7441b375cf15a4b04579cbae82d1db516d2fed8da6062b4e5e06610 not found: ID does not exist" containerID="560ff3c9b7441b375cf15a4b04579cbae82d1db516d2fed8da6062b4e5e06610"
Jul 30 00:19:38.945604 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.945534    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560ff3c9b7441b375cf15a4b04579cbae82d1db516d2fed8da6062b4e5e06610"} err="failed to get container status \"560ff3c9b7441b375cf15a4b04579cbae82d1db516d2fed8da6062b4e5e06610\": rpc error: code = NotFound desc = could not find container \"560ff3c9b7441b375cf15a4b04579cbae82d1db516d2fed8da6062b4e5e06610\": container with ID starting with 560ff3c9b7441b375cf15a4b04579cbae82d1db516d2fed8da6062b4e5e06610 not found: ID does not exist"
Jul 30 00:19:38.945604 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.945548    2579 scope.go:117] "RemoveContainer" containerID="50d276187536202cd8838a2115e87582861db8cb14a58c4d021ae7c861bd9467"
Jul 30 00:19:38.945863 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:19:38.945793    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50d276187536202cd8838a2115e87582861db8cb14a58c4d021ae7c861bd9467\": container with ID starting with 50d276187536202cd8838a2115e87582861db8cb14a58c4d021ae7c861bd9467 not found: ID does not exist" containerID="50d276187536202cd8838a2115e87582861db8cb14a58c4d021ae7c861bd9467"
Jul 30 00:19:38.945863 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.945819    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d276187536202cd8838a2115e87582861db8cb14a58c4d021ae7c861bd9467"} err="failed to get container status \"50d276187536202cd8838a2115e87582861db8cb14a58c4d021ae7c861bd9467\": rpc error: code = NotFound desc = could not find container \"50d276187536202cd8838a2115e87582861db8cb14a58c4d021ae7c861bd9467\": container with ID starting with 50d276187536202cd8838a2115e87582861db8cb14a58c4d021ae7c861bd9467 not found: ID does not exist"
Jul 30 00:19:38.945863 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.945841    2579 scope.go:117] "RemoveContainer" containerID="e6917acfa96fe1966c6e0c95ce953cc1afa534b54f15490993b77f4a80a6ac8b"
Jul 30 00:19:38.946116 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:19:38.946086    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6917acfa96fe1966c6e0c95ce953cc1afa534b54f15490993b77f4a80a6ac8b\": container with ID starting with e6917acfa96fe1966c6e0c95ce953cc1afa534b54f15490993b77f4a80a6ac8b not found: ID does not exist" containerID="e6917acfa96fe1966c6e0c95ce953cc1afa534b54f15490993b77f4a80a6ac8b"
Jul 30 00:19:38.946189 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.946109    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6917acfa96fe1966c6e0c95ce953cc1afa534b54f15490993b77f4a80a6ac8b"} err="failed to get container status \"e6917acfa96fe1966c6e0c95ce953cc1afa534b54f15490993b77f4a80a6ac8b\": rpc error: code = NotFound desc = could not find container \"e6917acfa96fe1966c6e0c95ce953cc1afa534b54f15490993b77f4a80a6ac8b\": container with ID starting with e6917acfa96fe1966c6e0c95ce953cc1afa534b54f15490993b77f4a80a6ac8b not found: ID does not exist"
Jul 30 00:19:38.946189 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.946131    2579 scope.go:117] "RemoveContainer" containerID="c68bbf17bce8523d10159185f16bc694efd668aef8f046069016ad4102c73a66"
Jul 30 00:19:38.946445 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:19:38.946428    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c68bbf17bce8523d10159185f16bc694efd668aef8f046069016ad4102c73a66\": container with ID starting with c68bbf17bce8523d10159185f16bc694efd668aef8f046069016ad4102c73a66 not found: ID does not exist" containerID="c68bbf17bce8523d10159185f16bc694efd668aef8f046069016ad4102c73a66"
Jul 30 00:19:38.946491 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.946449    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c68bbf17bce8523d10159185f16bc694efd668aef8f046069016ad4102c73a66"} err="failed to get container status \"c68bbf17bce8523d10159185f16bc694efd668aef8f046069016ad4102c73a66\": rpc error: code = NotFound desc = could not find container \"c68bbf17bce8523d10159185f16bc694efd668aef8f046069016ad4102c73a66\": container with ID starting with c68bbf17bce8523d10159185f16bc694efd668aef8f046069016ad4102c73a66 not found: ID does not exist"
Jul 30 00:19:38.946491 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.946463    2579 scope.go:117] "RemoveContainer" containerID="400be706d0bad2580b8b98036a1156cbf3b3a542372701d692045235ec922061"
Jul 30 00:19:38.946699 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:19:38.946686    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400be706d0bad2580b8b98036a1156cbf3b3a542372701d692045235ec922061\": container with ID starting with 400be706d0bad2580b8b98036a1156cbf3b3a542372701d692045235ec922061 not found: ID does not exist" containerID="400be706d0bad2580b8b98036a1156cbf3b3a542372701d692045235ec922061"
Jul 30 00:19:38.946742 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.946703    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400be706d0bad2580b8b98036a1156cbf3b3a542372701d692045235ec922061"} err="failed to get container status \"400be706d0bad2580b8b98036a1156cbf3b3a542372701d692045235ec922061\": rpc error: code = NotFound desc = could not find container \"400be706d0bad2580b8b98036a1156cbf3b3a542372701d692045235ec922061\": container with ID starting with 400be706d0bad2580b8b98036a1156cbf3b3a542372701d692045235ec922061 not found: ID does not exist"
Jul 30 00:19:38.946742 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.946715    2579 scope.go:117] "RemoveContainer" containerID="2f140f96642b0c5aa1503411fe451d423c257dfd7ab1e1452506e8bd9f250661"
Jul 30 00:19:38.946961 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:19:38.946944    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f140f96642b0c5aa1503411fe451d423c257dfd7ab1e1452506e8bd9f250661\": container with ID starting with 2f140f96642b0c5aa1503411fe451d423c257dfd7ab1e1452506e8bd9f250661 not found: ID does not exist" containerID="2f140f96642b0c5aa1503411fe451d423c257dfd7ab1e1452506e8bd9f250661"
Jul 30 00:19:38.947020 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.946964    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f140f96642b0c5aa1503411fe451d423c257dfd7ab1e1452506e8bd9f250661"} err="failed to get container status \"2f140f96642b0c5aa1503411fe451d423c257dfd7ab1e1452506e8bd9f250661\": rpc error: code = NotFound desc = could not find container \"2f140f96642b0c5aa1503411fe451d423c257dfd7ab1e1452506e8bd9f250661\": container with ID starting with 2f140f96642b0c5aa1503411fe451d423c257dfd7ab1e1452506e8bd9f250661 not found: ID does not exist"
Jul 30 00:19:38.947020 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.946978    2579 scope.go:117] "RemoveContainer" containerID="1478560230ebfce046448e3e4e3dda3e275f8902ac84f95d1509c0071ad8c55e"
Jul 30 00:19:38.947197 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.947180    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1478560230ebfce046448e3e4e3dda3e275f8902ac84f95d1509c0071ad8c55e"} err="failed to get container status \"1478560230ebfce046448e3e4e3dda3e275f8902ac84f95d1509c0071ad8c55e\": rpc error: code = NotFound desc = could not find container \"1478560230ebfce046448e3e4e3dda3e275f8902ac84f95d1509c0071ad8c55e\": container with ID starting with 1478560230ebfce046448e3e4e3dda3e275f8902ac84f95d1509c0071ad8c55e not found: ID does not exist"
Jul 30 00:19:38.947259 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.947198    2579 scope.go:117] "RemoveContainer" containerID="560ff3c9b7441b375cf15a4b04579cbae82d1db516d2fed8da6062b4e5e06610"
Jul 30 00:19:38.947473 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.947450    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560ff3c9b7441b375cf15a4b04579cbae82d1db516d2fed8da6062b4e5e06610"} err="failed to get container status \"560ff3c9b7441b375cf15a4b04579cbae82d1db516d2fed8da6062b4e5e06610\": rpc error: code = NotFound desc = could not find container \"560ff3c9b7441b375cf15a4b04579cbae82d1db516d2fed8da6062b4e5e06610\": container with ID starting with 560ff3c9b7441b375cf15a4b04579cbae82d1db516d2fed8da6062b4e5e06610 not found: ID does not exist"
Jul 30 00:19:38.947536 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.947479    2579 scope.go:117] "RemoveContainer" containerID="50d276187536202cd8838a2115e87582861db8cb14a58c4d021ae7c861bd9467"
Jul 30 00:19:38.947715 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.947699    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d276187536202cd8838a2115e87582861db8cb14a58c4d021ae7c861bd9467"} err="failed to get container status \"50d276187536202cd8838a2115e87582861db8cb14a58c4d021ae7c861bd9467\": rpc error: code = NotFound desc = could not find container \"50d276187536202cd8838a2115e87582861db8cb14a58c4d021ae7c861bd9467\": container with ID starting with 50d276187536202cd8838a2115e87582861db8cb14a58c4d021ae7c861bd9467 not found: ID does not exist"
Jul 30 00:19:38.947781 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.947717    2579 scope.go:117] "RemoveContainer" containerID="e6917acfa96fe1966c6e0c95ce953cc1afa534b54f15490993b77f4a80a6ac8b"
Jul 30 00:19:38.947943 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.947924    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6917acfa96fe1966c6e0c95ce953cc1afa534b54f15490993b77f4a80a6ac8b"} err="failed to get container status \"e6917acfa96fe1966c6e0c95ce953cc1afa534b54f15490993b77f4a80a6ac8b\": rpc error: code = NotFound desc = could not find container \"e6917acfa96fe1966c6e0c95ce953cc1afa534b54f15490993b77f4a80a6ac8b\": container with ID starting with e6917acfa96fe1966c6e0c95ce953cc1afa534b54f15490993b77f4a80a6ac8b not found: ID does not exist"
Jul 30 00:19:38.947989 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.947945    2579 scope.go:117] "RemoveContainer" containerID="c68bbf17bce8523d10159185f16bc694efd668aef8f046069016ad4102c73a66"
Jul 30 00:19:38.948158 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.948138    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c68bbf17bce8523d10159185f16bc694efd668aef8f046069016ad4102c73a66"} err="failed to get container status \"c68bbf17bce8523d10159185f16bc694efd668aef8f046069016ad4102c73a66\": rpc error: code = NotFound desc = could not find container \"c68bbf17bce8523d10159185f16bc694efd668aef8f046069016ad4102c73a66\": container with ID starting with c68bbf17bce8523d10159185f16bc694efd668aef8f046069016ad4102c73a66 not found: ID does not exist"
Jul 30 00:19:38.948203 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.948158    2579 scope.go:117] "RemoveContainer" containerID="400be706d0bad2580b8b98036a1156cbf3b3a542372701d692045235ec922061"
Jul 30 00:19:38.948470 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.948455    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400be706d0bad2580b8b98036a1156cbf3b3a542372701d692045235ec922061"} err="failed to get container status \"400be706d0bad2580b8b98036a1156cbf3b3a542372701d692045235ec922061\": rpc error: code = NotFound desc = could not find container \"400be706d0bad2580b8b98036a1156cbf3b3a542372701d692045235ec922061\": container with ID starting with 400be706d0bad2580b8b98036a1156cbf3b3a542372701d692045235ec922061 not found: ID does not exist"
Jul 30 00:19:38.948470 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.948469    2579 scope.go:117] "RemoveContainer" containerID="2f140f96642b0c5aa1503411fe451d423c257dfd7ab1e1452506e8bd9f250661"
Jul 30 00:19:38.948688 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.948667    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f140f96642b0c5aa1503411fe451d423c257dfd7ab1e1452506e8bd9f250661"} err="failed to get container status \"2f140f96642b0c5aa1503411fe451d423c257dfd7ab1e1452506e8bd9f250661\": rpc error: code = NotFound desc = could not find container \"2f140f96642b0c5aa1503411fe451d423c257dfd7ab1e1452506e8bd9f250661\": container with ID starting with 2f140f96642b0c5aa1503411fe451d423c257dfd7ab1e1452506e8bd9f250661 not found: ID does not exist"
Jul 30 00:19:38.979383 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.979317    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 00:19:38.979498 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.979452    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:38.982216 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.982188    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 30 00:19:38.982338 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.982258    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-j4gt8\""
Jul 30 00:19:38.982338 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.982274    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 30 00:19:38.982451 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.982361    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 30 00:19:38.982451 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.982199    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 30 00:19:38.982665 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.982608    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 30 00:19:38.982665 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.982656    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 30 00:19:38.982815 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.982686    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 30 00:19:38.982966 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.982948    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 30 00:19:38.986413 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:38.986393    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 30 00:19:39.115448 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.115418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.115608 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.115464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.115608 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.115492    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f26fbc8a-a80a-4d45-996f-d26e043a8627-config-out\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.115608 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.115518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-config-volume\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.115608 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.115541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.115777 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.115605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxpwk\" (UniqueName: \"kubernetes.io/projected/f26fbc8a-a80a-4d45-996f-d26e043a8627-kube-api-access-kxpwk\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.115777 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.115635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.115777 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.115673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f26fbc8a-a80a-4d45-996f-d26e043a8627-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.115777 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.115726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f26fbc8a-a80a-4d45-996f-d26e043a8627-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.115902 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.115780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f26fbc8a-a80a-4d45-996f-d26e043a8627-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.115902 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.115804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.115902 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.115831    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-web-config\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.115902 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.115851    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f26fbc8a-a80a-4d45-996f-d26e043a8627-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.216548 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.216517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.216548 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.216548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f26fbc8a-a80a-4d45-996f-d26e043a8627-config-out\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.216760 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.216569    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-config-volume\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.216760 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.216585    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.216760 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.216623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxpwk\" (UniqueName: \"kubernetes.io/projected/f26fbc8a-a80a-4d45-996f-d26e043a8627-kube-api-access-kxpwk\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.216760 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.216647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.216760 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.216686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f26fbc8a-a80a-4d45-996f-d26e043a8627-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.216760 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.216711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f26fbc8a-a80a-4d45-996f-d26e043a8627-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.216760 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.216741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f26fbc8a-a80a-4d45-996f-d26e043a8627-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.217102 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.216768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.217102 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.216801    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-web-config\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.217102 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.216826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f26fbc8a-a80a-4d45-996f-d26e043a8627-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.217102 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.216867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.217329 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.217172    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f26fbc8a-a80a-4d45-996f-d26e043a8627-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.218093 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.217761    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f26fbc8a-a80a-4d45-996f-d26e043a8627-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.219460 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.219339    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f26fbc8a-a80a-4d45-996f-d26e043a8627-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.219667 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.219633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.219730 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.219704    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.220079 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.220054    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.220190 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.220103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-web-config\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.220190 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.220168    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-config-volume\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.220319 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.220197    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.220711 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.220685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f26fbc8a-a80a-4d45-996f-d26e043a8627-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.220789 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.220726    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f26fbc8a-a80a-4d45-996f-d26e043a8627-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.221173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.221157    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f26fbc8a-a80a-4d45-996f-d26e043a8627-config-out\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.224676 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.224658    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxpwk\" (UniqueName: \"kubernetes.io/projected/f26fbc8a-a80a-4d45-996f-d26e043a8627-kube-api-access-kxpwk\") pod \"alertmanager-main-0\" (UID: \"f26fbc8a-a80a-4d45-996f-d26e043a8627\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.290856 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.290798    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 30 00:19:39.321673 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.321647    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52138b7c-2f87-486d-855c-dc6bb4c8c39b" path="/var/lib/kubelet/pods/52138b7c-2f87-486d-855c-dc6bb4c8c39b/volumes"
Jul 30 00:19:39.452402 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.452378    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 00:19:39.454525 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:19:39.454497    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26fbc8a_a80a_4d45_996f_d26e043a8627.slice/crio-82a6daf1cd714e7c5dd09ba653c609fbcb2e2af26e84f20b632c3d5df2c3b466 WatchSource:0}: Error finding container 82a6daf1cd714e7c5dd09ba653c609fbcb2e2af26e84f20b632c3d5df2c3b466: Status 404 returned error can't find the container with id 82a6daf1cd714e7c5dd09ba653c609fbcb2e2af26e84f20b632c3d5df2c3b466
Jul 30 00:19:39.897633 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.897598    2579 generic.go:356] "Generic (PLEG): container finished" podID="f26fbc8a-a80a-4d45-996f-d26e043a8627" containerID="2746f06fa46b7aaba69da7c49af89e516a78a3fe34445b40d8da1bb8af345d5c" exitCode=0
Jul 30 00:19:39.898066 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.897677    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f26fbc8a-a80a-4d45-996f-d26e043a8627","Type":"ContainerDied","Data":"2746f06fa46b7aaba69da7c49af89e516a78a3fe34445b40d8da1bb8af345d5c"}
Jul 30 00:19:39.898066 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:39.897747    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f26fbc8a-a80a-4d45-996f-d26e043a8627","Type":"ContainerStarted","Data":"82a6daf1cd714e7c5dd09ba653c609fbcb2e2af26e84f20b632c3d5df2c3b466"}
Jul 30 00:19:40.906673 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:40.906631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f26fbc8a-a80a-4d45-996f-d26e043a8627","Type":"ContainerStarted","Data":"7b4dc666030e5f041b7c54c002c2637c86a430707797544b6c19bdf11a5d03b0"}
Jul 30 00:19:40.907065 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:40.906720    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f26fbc8a-a80a-4d45-996f-d26e043a8627","Type":"ContainerStarted","Data":"9c547149b31c46c1232a9d4a6d151d8b9044347fdcc5be9ad686254260c3be60"}
Jul 30 00:19:40.907065 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:40.906761    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f26fbc8a-a80a-4d45-996f-d26e043a8627","Type":"ContainerStarted","Data":"f4ac410001dcd64c2617716dde85c1d744b2e3417160caa13e137ac219234f89"}
Jul 30 00:19:40.907065 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:40.906799    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f26fbc8a-a80a-4d45-996f-d26e043a8627","Type":"ContainerStarted","Data":"c6a5d4bb82730526042cd53575baafad79d7c470b0f2fb0f29c00b8c068a5dda"}
Jul 30 00:19:40.907065 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:40.906839    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f26fbc8a-a80a-4d45-996f-d26e043a8627","Type":"ContainerStarted","Data":"2a2cffa7a0743c96f9c94cc25087640e3b9a0f83a12b46c10e6d86a2321b36e6"}
Jul 30 00:19:40.907065 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:40.906872    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f26fbc8a-a80a-4d45-996f-d26e043a8627","Type":"ContainerStarted","Data":"87fd576abd4bca6af819e6168777a09f92cdcb19dee2a8c3c26786758e253e96"}
Jul 30 00:19:40.933694 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:40.933647    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.910362557 podStartE2EDuration="2.933631554s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.268997ms" isStatefulPod=false podCreationTimestamp="2026-07-30 00:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:19:40.931044647 +0000 UTC m=+106.228311592" watchObservedRunningTime="2026-07-30 00:19:40.933631554 +0000 UTC m=+106.230898485"
Jul 30 00:19:41.545826 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:41.545782    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 00:19:41.546177 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:41.546141    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="prometheus" containerID="cri-o://e5454c4550a85b91a8c6588292e35f8f7b21590debf8cd49750b5ddbe9072477" gracePeriod=600
Jul 30 00:19:41.546331 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:41.546232    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="config-reloader" containerID="cri-o://0162c846c23b5cad1f94fac39edc2cf93ca2d87a48829ea5c51bf17a8e5c1f72" gracePeriod=600
Jul 30 00:19:41.546331 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:41.546152    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="thanos-sidecar" containerID="cri-o://3f439dd127c49f058eda08eb41eab0759c1a0df4bd8b1d975d52c819f0d00e17" gracePeriod=600
Jul 30 00:19:41.546458 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:41.546168    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="kube-rbac-proxy" containerID="cri-o://8a67756d1e075940c3bf5f8a3f0c3d1a8819fa26975900e1214f2fb6c4783346" gracePeriod=600
Jul 30 00:19:41.546458 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:41.546173    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="kube-rbac-proxy-web" containerID="cri-o://420e56456e90504ae62fe3f9aa0a127028363554fd3e12743ade7c7f5350feeb" gracePeriod=600
Jul 30 00:19:41.546458 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:41.546199    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://dc32b0fc6601fdb6c158e50a53384261c3334593c0b82ae31942e81bbdca22d6" gracePeriod=600
Jul 30 00:19:41.914832 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:41.914796    2579 generic.go:356] "Generic (PLEG): container finished" podID="8c89a159-e52a-4274-b236-131ecb0128e8" containerID="dc32b0fc6601fdb6c158e50a53384261c3334593c0b82ae31942e81bbdca22d6" exitCode=0
Jul 30 00:19:41.914832 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:41.914822    2579 generic.go:356] "Generic (PLEG): container finished" podID="8c89a159-e52a-4274-b236-131ecb0128e8" containerID="8a67756d1e075940c3bf5f8a3f0c3d1a8819fa26975900e1214f2fb6c4783346" exitCode=0
Jul 30 00:19:41.914832 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:41.914828    2579 generic.go:356] "Generic (PLEG): container finished" podID="8c89a159-e52a-4274-b236-131ecb0128e8" containerID="3f439dd127c49f058eda08eb41eab0759c1a0df4bd8b1d975d52c819f0d00e17" exitCode=0
Jul 30 00:19:41.914832 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:41.914835    2579 generic.go:356] "Generic (PLEG): container finished" podID="8c89a159-e52a-4274-b236-131ecb0128e8" containerID="0162c846c23b5cad1f94fac39edc2cf93ca2d87a48829ea5c51bf17a8e5c1f72" exitCode=0
Jul 30 00:19:41.914832 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:41.914841    2579 generic.go:356] "Generic (PLEG): container finished" podID="8c89a159-e52a-4274-b236-131ecb0128e8" containerID="e5454c4550a85b91a8c6588292e35f8f7b21590debf8cd49750b5ddbe9072477" exitCode=0
Jul 30 00:19:41.915391 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:41.914865    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c89a159-e52a-4274-b236-131ecb0128e8","Type":"ContainerDied","Data":"dc32b0fc6601fdb6c158e50a53384261c3334593c0b82ae31942e81bbdca22d6"}
Jul 30 00:19:41.915391 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:41.914939    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c89a159-e52a-4274-b236-131ecb0128e8","Type":"ContainerDied","Data":"8a67756d1e075940c3bf5f8a3f0c3d1a8819fa26975900e1214f2fb6c4783346"}
Jul 30 00:19:41.915391 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:41.914966    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c89a159-e52a-4274-b236-131ecb0128e8","Type":"ContainerDied","Data":"3f439dd127c49f058eda08eb41eab0759c1a0df4bd8b1d975d52c819f0d00e17"}
Jul 30 00:19:41.915391 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:41.914991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c89a159-e52a-4274-b236-131ecb0128e8","Type":"ContainerDied","Data":"0162c846c23b5cad1f94fac39edc2cf93ca2d87a48829ea5c51bf17a8e5c1f72"}
Jul 30 00:19:41.915391 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:41.915018    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c89a159-e52a-4274-b236-131ecb0128e8","Type":"ContainerDied","Data":"e5454c4550a85b91a8c6588292e35f8f7b21590debf8cd49750b5ddbe9072477"}
Jul 30 00:19:42.782314 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.782291    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:42.850092 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850058    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-prometheus-k8s-tls\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.850092 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850097    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-web-config\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.850309 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850127    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-configmap-kubelet-serving-ca-bundle\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.850309 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850152    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.850309 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850183    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-metrics-client-certs\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.850309 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850208    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-rulefiles-2\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.850309 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850239    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-config\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.850309 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850288    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-grpc-tls\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.850603 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850313    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-thanos-prometheus-http-client-file\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.850603 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850348    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c89a159-e52a-4274-b236-131ecb0128e8-config-out\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.850603 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850372    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-trusted-ca-bundle\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.850603 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850427    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-rulefiles-1\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.850603 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850447    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-configmap-metrics-client-ca\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.850603 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850478    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-rulefiles-0\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.850603 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850496    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c89a159-e52a-4274-b236-131ecb0128e8-tls-assets\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.850603 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850511    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-kube-rbac-proxy\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.850603 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850541    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5w8ml\" (UniqueName: \"kubernetes.io/projected/8c89a159-e52a-4274-b236-131ecb0128e8-kube-api-access-5w8ml\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.850603 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850560    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.850603 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850592    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-configmap-serving-certs-ca-bundle\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.850603 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850610    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-db\") pod \"8c89a159-e52a-4274-b236-131ecb0128e8\" (UID: \"8c89a159-e52a-4274-b236-131ecb0128e8\") "
Jul 30 00:19:42.851143 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.850992    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:19:42.851932 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.851897    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:19:42.852598 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.852383    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:19:42.853230 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.852894    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:19:42.853428 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.853233    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:19:42.853854 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.853640    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:19:42.853854 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.853676    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:19:42.853854 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.853716    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:19:42.854325 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.854285    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:19:42.857387 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.857334    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:19:42.857561 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.857425    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:19:42.857633 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.857585    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c89a159-e52a-4274-b236-131ecb0128e8-config-out" (OuterVolumeSpecName: "config-out") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:19:42.858700 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.858672    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:19:42.859425 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.859098    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c89a159-e52a-4274-b236-131ecb0128e8-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:19:42.859913 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.859878    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:19:42.860042 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.860019    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c89a159-e52a-4274-b236-131ecb0128e8-kube-api-access-5w8ml" (OuterVolumeSpecName: "kube-api-access-5w8ml") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "kube-api-access-5w8ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:19:42.860649 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.860582    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-config" (OuterVolumeSpecName: "config") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:19:42.860649 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.860627    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:19:42.863307 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.863280    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:19:42.870470 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.870434    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-web-config" (OuterVolumeSpecName: "web-config") pod "8c89a159-e52a-4274-b236-131ecb0128e8" (UID: "8c89a159-e52a-4274-b236-131ecb0128e8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:19:42.923080 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.923042    2579 generic.go:356] "Generic (PLEG): container finished" podID="8c89a159-e52a-4274-b236-131ecb0128e8" containerID="420e56456e90504ae62fe3f9aa0a127028363554fd3e12743ade7c7f5350feeb" exitCode=0
Jul 30 00:19:42.923618 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.923090    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c89a159-e52a-4274-b236-131ecb0128e8","Type":"ContainerDied","Data":"420e56456e90504ae62fe3f9aa0a127028363554fd3e12743ade7c7f5350feeb"}
Jul 30 00:19:42.923618 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.923159    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:42.923618 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.923197    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c89a159-e52a-4274-b236-131ecb0128e8","Type":"ContainerDied","Data":"8d982dd04ac4797639cc6a48e9f471fbe8c63c974ef1a0a674532cca9c2a8c6b"}
Jul 30 00:19:42.923618 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.923262    2579 scope.go:117] "RemoveContainer" containerID="dc32b0fc6601fdb6c158e50a53384261c3334593c0b82ae31942e81bbdca22d6"
Jul 30 00:19:42.934892 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.934873    2579 scope.go:117] "RemoveContainer" containerID="8a67756d1e075940c3bf5f8a3f0c3d1a8819fa26975900e1214f2fb6c4783346"
Jul 30 00:19:42.942409 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.942386    2579 scope.go:117] "RemoveContainer" containerID="420e56456e90504ae62fe3f9aa0a127028363554fd3e12743ade7c7f5350feeb"
Jul 30 00:19:42.950119 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.950096    2579 scope.go:117] "RemoveContainer" containerID="3f439dd127c49f058eda08eb41eab0759c1a0df4bd8b1d975d52c819f0d00e17"
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951214    2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-grpc-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951271    2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951289    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c89a159-e52a-4274-b236-131ecb0128e8-config-out\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951306    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951320    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951335    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-configmap-metrics-client-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951348    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951361    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c89a159-e52a-4274-b236-131ecb0128e8-tls-assets\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951375    2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-kube-rbac-proxy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951387    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w8ml\" (UniqueName: \"kubernetes.io/projected/8c89a159-e52a-4274-b236-131ecb0128e8-kube-api-access-5w8ml\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951402    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951416    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951429    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-db\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951441    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951455    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-web-config\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951467    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951480    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951493    2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-secret-metrics-client-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951507    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c89a159-e52a-4274-b236-131ecb0128e8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.951237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.951522    2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c89a159-e52a-4274-b236-131ecb0128e8-config\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:19:42.952390 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.952179    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 00:19:42.956678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.956653    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 00:19:42.958560 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.958541    2579 scope.go:117] "RemoveContainer" containerID="0162c846c23b5cad1f94fac39edc2cf93ca2d87a48829ea5c51bf17a8e5c1f72"
Jul 30 00:19:42.966771 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.966752    2579 scope.go:117] "RemoveContainer" containerID="e5454c4550a85b91a8c6588292e35f8f7b21590debf8cd49750b5ddbe9072477"
Jul 30 00:19:42.974097 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.974081    2579 scope.go:117] "RemoveContainer" containerID="777b3ac55e59c8a495a8c68c36208b1426029f1df494b171e1e807ae1d2f105a"
Jul 30 00:19:42.981796 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.981778    2579 scope.go:117] "RemoveContainer" containerID="dc32b0fc6601fdb6c158e50a53384261c3334593c0b82ae31942e81bbdca22d6"
Jul 30 00:19:42.982075 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:19:42.982052    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc32b0fc6601fdb6c158e50a53384261c3334593c0b82ae31942e81bbdca22d6\": container with ID starting with dc32b0fc6601fdb6c158e50a53384261c3334593c0b82ae31942e81bbdca22d6 not found: ID does not exist" containerID="dc32b0fc6601fdb6c158e50a53384261c3334593c0b82ae31942e81bbdca22d6"
Jul 30 00:19:42.982126 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.982087    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc32b0fc6601fdb6c158e50a53384261c3334593c0b82ae31942e81bbdca22d6"} err="failed to get container status \"dc32b0fc6601fdb6c158e50a53384261c3334593c0b82ae31942e81bbdca22d6\": rpc error: code = NotFound desc = could not find container \"dc32b0fc6601fdb6c158e50a53384261c3334593c0b82ae31942e81bbdca22d6\": container with ID starting with dc32b0fc6601fdb6c158e50a53384261c3334593c0b82ae31942e81bbdca22d6 not found: ID does not exist"
Jul 30 00:19:42.982126 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.982117    2579 scope.go:117] "RemoveContainer" containerID="8a67756d1e075940c3bf5f8a3f0c3d1a8819fa26975900e1214f2fb6c4783346"
Jul 30 00:19:42.982429 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:19:42.982412    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a67756d1e075940c3bf5f8a3f0c3d1a8819fa26975900e1214f2fb6c4783346\": container with ID starting with 8a67756d1e075940c3bf5f8a3f0c3d1a8819fa26975900e1214f2fb6c4783346 not found: ID does not exist" containerID="8a67756d1e075940c3bf5f8a3f0c3d1a8819fa26975900e1214f2fb6c4783346"
Jul 30 00:19:42.982480 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.982442    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a67756d1e075940c3bf5f8a3f0c3d1a8819fa26975900e1214f2fb6c4783346"} err="failed to get container status \"8a67756d1e075940c3bf5f8a3f0c3d1a8819fa26975900e1214f2fb6c4783346\": rpc error: code = NotFound desc = could not find container \"8a67756d1e075940c3bf5f8a3f0c3d1a8819fa26975900e1214f2fb6c4783346\": container with ID starting with 8a67756d1e075940c3bf5f8a3f0c3d1a8819fa26975900e1214f2fb6c4783346 not found: ID does not exist"
Jul 30 00:19:42.982518 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.982484    2579 scope.go:117] "RemoveContainer" containerID="420e56456e90504ae62fe3f9aa0a127028363554fd3e12743ade7c7f5350feeb"
Jul 30 00:19:42.982563 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.982545    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 00:19:42.982788 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:19:42.982769    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"420e56456e90504ae62fe3f9aa0a127028363554fd3e12743ade7c7f5350feeb\": container with ID starting with 420e56456e90504ae62fe3f9aa0a127028363554fd3e12743ade7c7f5350feeb not found: ID does not exist" containerID="420e56456e90504ae62fe3f9aa0a127028363554fd3e12743ade7c7f5350feeb"
Jul 30 00:19:42.982877 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.982795    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"420e56456e90504ae62fe3f9aa0a127028363554fd3e12743ade7c7f5350feeb"} err="failed to get container status \"420e56456e90504ae62fe3f9aa0a127028363554fd3e12743ade7c7f5350feeb\": rpc error: code = NotFound desc = could not find container \"420e56456e90504ae62fe3f9aa0a127028363554fd3e12743ade7c7f5350feeb\": container with ID starting with 420e56456e90504ae62fe3f9aa0a127028363554fd3e12743ade7c7f5350feeb not found: ID does not exist"
Jul 30 00:19:42.982877 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.982818    2579 scope.go:117] "RemoveContainer" containerID="3f439dd127c49f058eda08eb41eab0759c1a0df4bd8b1d975d52c819f0d00e17"
Jul 30 00:19:42.982961 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.982888    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="kube-rbac-proxy-web"
Jul 30 00:19:42.982961 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.982906    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="kube-rbac-proxy-web"
Jul 30 00:19:42.982961 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.982919    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="config-reloader"
Jul 30 00:19:42.982961 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.982926    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="config-reloader"
Jul 30 00:19:42.982961 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.982932    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="kube-rbac-proxy"
Jul 30 00:19:42.982961 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.982937    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="kube-rbac-proxy"
Jul 30 00:19:42.982961 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.982947    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="prometheus"
Jul 30 00:19:42.982961 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.982954    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="prometheus"
Jul 30 00:19:42.983238 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.982970    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="init-config-reloader"
Jul 30 00:19:42.983238 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.982978    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="init-config-reloader"
Jul 30 00:19:42.983238 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.982989    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="thanos-sidecar"
Jul 30 00:19:42.983238 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.982997    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="thanos-sidecar"
Jul 30 00:19:42.983238 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.983009    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="kube-rbac-proxy-thanos"
Jul 30 00:19:42.983238 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.983014    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="kube-rbac-proxy-thanos"
Jul 30 00:19:42.983238 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:19:42.983111    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f439dd127c49f058eda08eb41eab0759c1a0df4bd8b1d975d52c819f0d00e17\": container with ID starting with 3f439dd127c49f058eda08eb41eab0759c1a0df4bd8b1d975d52c819f0d00e17 not found: ID does not exist" containerID="3f439dd127c49f058eda08eb41eab0759c1a0df4bd8b1d975d52c819f0d00e17"
Jul 30 00:19:42.983238 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.983122    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="kube-rbac-proxy-web"
Jul 30 00:19:42.983238 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.983133    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="kube-rbac-proxy"
Jul 30 00:19:42.983238 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.983132    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f439dd127c49f058eda08eb41eab0759c1a0df4bd8b1d975d52c819f0d00e17"} err="failed to get container status \"3f439dd127c49f058eda08eb41eab0759c1a0df4bd8b1d975d52c819f0d00e17\": rpc error: code = NotFound desc = could not find container \"3f439dd127c49f058eda08eb41eab0759c1a0df4bd8b1d975d52c819f0d00e17\": container with ID starting with 3f439dd127c49f058eda08eb41eab0759c1a0df4bd8b1d975d52c819f0d00e17 not found: ID does not exist"
Jul 30 00:19:42.983238 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.983151    2579 scope.go:117] "RemoveContainer" containerID="0162c846c23b5cad1f94fac39edc2cf93ca2d87a48829ea5c51bf17a8e5c1f72"
Jul 30 00:19:42.983238 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.983141    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="thanos-sidecar"
Jul 30 00:19:42.983238 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.983206    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="prometheus"
Jul 30 00:19:42.983238 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.983214    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="kube-rbac-proxy-thanos"
Jul 30 00:19:42.983238 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.983221    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" containerName="config-reloader"
Jul 30 00:19:42.983764 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:19:42.983436    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0162c846c23b5cad1f94fac39edc2cf93ca2d87a48829ea5c51bf17a8e5c1f72\": container with ID starting with 0162c846c23b5cad1f94fac39edc2cf93ca2d87a48829ea5c51bf17a8e5c1f72 not found: ID does not exist" containerID="0162c846c23b5cad1f94fac39edc2cf93ca2d87a48829ea5c51bf17a8e5c1f72"
Jul 30 00:19:42.983764 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.983466    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0162c846c23b5cad1f94fac39edc2cf93ca2d87a48829ea5c51bf17a8e5c1f72"} err="failed to get container status \"0162c846c23b5cad1f94fac39edc2cf93ca2d87a48829ea5c51bf17a8e5c1f72\": rpc error: code = NotFound desc = could not find container \"0162c846c23b5cad1f94fac39edc2cf93ca2d87a48829ea5c51bf17a8e5c1f72\": container with ID starting with 0162c846c23b5cad1f94fac39edc2cf93ca2d87a48829ea5c51bf17a8e5c1f72 not found: ID does not exist"
Jul 30 00:19:42.983764 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.983489    2579 scope.go:117] "RemoveContainer" containerID="e5454c4550a85b91a8c6588292e35f8f7b21590debf8cd49750b5ddbe9072477"
Jul 30 00:19:42.983888 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:19:42.983760    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5454c4550a85b91a8c6588292e35f8f7b21590debf8cd49750b5ddbe9072477\": container with ID starting with e5454c4550a85b91a8c6588292e35f8f7b21590debf8cd49750b5ddbe9072477 not found: ID does not exist" containerID="e5454c4550a85b91a8c6588292e35f8f7b21590debf8cd49750b5ddbe9072477"
Jul 30 00:19:42.983888 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.983789    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5454c4550a85b91a8c6588292e35f8f7b21590debf8cd49750b5ddbe9072477"} err="failed to get container status \"e5454c4550a85b91a8c6588292e35f8f7b21590debf8cd49750b5ddbe9072477\": rpc error: code = NotFound desc = could not find container \"e5454c4550a85b91a8c6588292e35f8f7b21590debf8cd49750b5ddbe9072477\": container with ID starting with e5454c4550a85b91a8c6588292e35f8f7b21590debf8cd49750b5ddbe9072477 not found: ID does not exist"
Jul 30 00:19:42.983888 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.983813    2579 scope.go:117] "RemoveContainer" containerID="777b3ac55e59c8a495a8c68c36208b1426029f1df494b171e1e807ae1d2f105a"
Jul 30 00:19:42.984125 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:19:42.984098    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777b3ac55e59c8a495a8c68c36208b1426029f1df494b171e1e807ae1d2f105a\": container with ID starting with 777b3ac55e59c8a495a8c68c36208b1426029f1df494b171e1e807ae1d2f105a not found: ID does not exist" containerID="777b3ac55e59c8a495a8c68c36208b1426029f1df494b171e1e807ae1d2f105a"
Jul 30 00:19:42.984203 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.984133    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777b3ac55e59c8a495a8c68c36208b1426029f1df494b171e1e807ae1d2f105a"} err="failed to get container status \"777b3ac55e59c8a495a8c68c36208b1426029f1df494b171e1e807ae1d2f105a\": rpc error: code = NotFound desc = could not find container \"777b3ac55e59c8a495a8c68c36208b1426029f1df494b171e1e807ae1d2f105a\": container with ID starting with 777b3ac55e59c8a495a8c68c36208b1426029f1df494b171e1e807ae1d2f105a not found: ID does not exist"
Jul 30 00:19:42.988159 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.988141    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:42.992095 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.992070    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 30 00:19:42.992873 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.992662    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 30 00:19:42.992873 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.992677    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 30 00:19:42.992873 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.992729    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 30 00:19:42.992873 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.992740    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3vugdlve7uhrl\""
Jul 30 00:19:42.992873 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.992677    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 30 00:19:42.993448 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.993430    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 30 00:19:42.994386 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.994210    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 30 00:19:42.994386 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.994220    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 30 00:19:42.994386 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.994339    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 30 00:19:42.994386 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.994370    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 30 00:19:42.994609 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.994466    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 30 00:19:42.994742 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.994727    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rfzmp\""
Jul 30 00:19:42.995097 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.995077    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 30 00:19:42.996659 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.996635    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 30 00:19:42.998524 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:42.998504    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 30 00:19:43.006000 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.005975    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 00:19:43.052662 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.052623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.052662 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.052666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/789934fb-707f-4b9d-a425-92cfe9c3b66a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.052950 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.052707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.052950 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.052779    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/789934fb-707f-4b9d-a425-92cfe9c3b66a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.052950 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.052821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.052950 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.052853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.052950 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.052880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.052950 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.052917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.052950 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.052941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.053187 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.052999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.053187 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.053073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.053187 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.053100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.053187 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.053125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.053187 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.053174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/789934fb-707f-4b9d-a425-92cfe9c3b66a-config-out\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.053405 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.053269    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-web-config\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.053405 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.053297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.053405 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.053327    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-config\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.053405 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.053362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpjm2\" (UniqueName: \"kubernetes.io/projected/789934fb-707f-4b9d-a425-92cfe9c3b66a-kube-api-access-mpjm2\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.053405 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.053390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.053564 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.053415    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.154052 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-web-config\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.154286 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.154286 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-config\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.154286 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpjm2\" (UniqueName: \"kubernetes.io/projected/789934fb-707f-4b9d-a425-92cfe9c3b66a-kube-api-access-mpjm2\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.154286 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.154286 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.154286 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.154286 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/789934fb-707f-4b9d-a425-92cfe9c3b66a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.154637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.154637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/789934fb-707f-4b9d-a425-92cfe9c3b66a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.154637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154356    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.154637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.154637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.154637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.154637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.154637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.154637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.154637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.154637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154610    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.155238 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/789934fb-707f-4b9d-a425-92cfe9c3b66a-config-out\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.155238 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.154709    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/789934fb-707f-4b9d-a425-92cfe9c3b66a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.156908 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.155448    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.156908 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.155706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.156908 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.156094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.158648 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.157213    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/789934fb-707f-4b9d-a425-92cfe9c3b66a-config-out\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.158648 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.157363    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.158648 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.157454    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-config\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.158648 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.157767    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.158648 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.157959    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.158648 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.157984    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.158648 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.158334    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.158648 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.158421    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.159125 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.158687    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.159183 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.159159    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-web-config\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.160068 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.160043    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.160287 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.160263    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/789934fb-707f-4b9d-a425-92cfe9c3b66a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.160375 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.160345    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/789934fb-707f-4b9d-a425-92cfe9c3b66a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.160720 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.160699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.161051 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.161028    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/789934fb-707f-4b9d-a425-92cfe9c3b66a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.162873 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.162847    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpjm2\" (UniqueName: \"kubernetes.io/projected/789934fb-707f-4b9d-a425-92cfe9c3b66a-kube-api-access-mpjm2\") pod \"prometheus-k8s-0\" (UID: \"789934fb-707f-4b9d-a425-92cfe9c3b66a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.300076 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.300037    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:19:43.322939 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.322906    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c89a159-e52a-4274-b236-131ecb0128e8" path="/var/lib/kubelet/pods/8c89a159-e52a-4274-b236-131ecb0128e8/volumes"
Jul 30 00:19:43.444091 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.443923    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 00:19:43.448512 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:19:43.448479    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789934fb_707f_4b9d_a425_92cfe9c3b66a.slice/crio-4f9a32d04ad8ed1b678828b395fe3ab0581ac976d1dc2b3fc6f6c187c770a1aa WatchSource:0}: Error finding container 4f9a32d04ad8ed1b678828b395fe3ab0581ac976d1dc2b3fc6f6c187c770a1aa: Status 404 returned error can't find the container with id 4f9a32d04ad8ed1b678828b395fe3ab0581ac976d1dc2b3fc6f6c187c770a1aa
Jul 30 00:19:43.929469 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.929433    2579 generic.go:356] "Generic (PLEG): container finished" podID="789934fb-707f-4b9d-a425-92cfe9c3b66a" containerID="9766f14130d045cfb504ff3294649c4a16ff081d52ebc0329afe4948464cebce" exitCode=0
Jul 30 00:19:43.929954 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.929510    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789934fb-707f-4b9d-a425-92cfe9c3b66a","Type":"ContainerDied","Data":"9766f14130d045cfb504ff3294649c4a16ff081d52ebc0329afe4948464cebce"}
Jul 30 00:19:43.929954 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:43.929589    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789934fb-707f-4b9d-a425-92cfe9c3b66a","Type":"ContainerStarted","Data":"4f9a32d04ad8ed1b678828b395fe3ab0581ac976d1dc2b3fc6f6c187c770a1aa"}
Jul 30 00:19:44.942314 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:44.942278    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789934fb-707f-4b9d-a425-92cfe9c3b66a","Type":"ContainerStarted","Data":"c425d09da162d114b4fc8ef1cf2850b54316e7d864deba950199e25a419f0a0d"}
Jul 30 00:19:44.942823 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:44.942357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789934fb-707f-4b9d-a425-92cfe9c3b66a","Type":"ContainerStarted","Data":"ce97a94b167e08773c190dd98e502f6ef4d2d71a1b98bec039f0f0cec730bd39"}
Jul 30 00:19:44.942823 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:44.942386    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789934fb-707f-4b9d-a425-92cfe9c3b66a","Type":"ContainerStarted","Data":"81b63919c09926f71253e90e438e837b1e98f0515b030237c52000f225d63e2e"}
Jul 30 00:19:44.942823 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:44.942417    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789934fb-707f-4b9d-a425-92cfe9c3b66a","Type":"ContainerStarted","Data":"5aab938e8589029ea9287796330d7bf2dc02d8a82ec4267a27d4b1891c76b9e3"}
Jul 30 00:19:44.942823 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:44.942444    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789934fb-707f-4b9d-a425-92cfe9c3b66a","Type":"ContainerStarted","Data":"c7529d88ef010e6ce77775f758806c74984c3e9fce30ffe02c3c0f5859d96b89"}
Jul 30 00:19:44.942823 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:44.942485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789934fb-707f-4b9d-a425-92cfe9c3b66a","Type":"ContainerStarted","Data":"5e026521f9dd018d9fdee5e41fff6c7c14d03fa22375a3bd9dc13b864c23c951"}
Jul 30 00:19:44.974268 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:44.972309    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.948358549 podStartE2EDuration="2.972288218s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.929669ms" isStatefulPod=false podCreationTimestamp="2026-07-30 00:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:19:44.970904988 +0000 UTC m=+110.268171920" watchObservedRunningTime="2026-07-30 00:19:44.972288218 +0000 UTC m=+110.269555151"
Jul 30 00:19:48.300327 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:19:48.300292    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:20:43.301270 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:20:43.301198    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:20:43.321055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:20:43.321032    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:20:44.178859 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:20:44.178833    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 00:22:55.118726 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:22:55.118697    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:22:55.125601 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:22:55.125582    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 30 00:22:55.127623 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:22:55.127593    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:23:11.377556 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:11.377523    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9pm9p"]
Jul 30 00:23:11.379867 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:11.379846    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9pm9p"
Jul 30 00:23:11.382472 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:11.382441    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 30 00:23:11.382592 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:11.382561    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 30 00:23:11.388758 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:11.388466    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9pm9p"]
Jul 30 00:23:11.505415 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:11.505380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d4107db-fc48-4469-9991-8fade0c98697-bound-sa-token\") pod \"cert-manager-55b4d954d5-9pm9p\" (UID: \"2d4107db-fc48-4469-9991-8fade0c98697\") " pod="cert-manager/cert-manager-55b4d954d5-9pm9p"
Jul 30 00:23:11.505596 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:11.505429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rck8s\" (UniqueName: \"kubernetes.io/projected/2d4107db-fc48-4469-9991-8fade0c98697-kube-api-access-rck8s\") pod \"cert-manager-55b4d954d5-9pm9p\" (UID: \"2d4107db-fc48-4469-9991-8fade0c98697\") " pod="cert-manager/cert-manager-55b4d954d5-9pm9p"
Jul 30 00:23:11.606269 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:11.606212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d4107db-fc48-4469-9991-8fade0c98697-bound-sa-token\") pod \"cert-manager-55b4d954d5-9pm9p\" (UID: \"2d4107db-fc48-4469-9991-8fade0c98697\") " pod="cert-manager/cert-manager-55b4d954d5-9pm9p"
Jul 30 00:23:11.606428 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:11.606283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rck8s\" (UniqueName: \"kubernetes.io/projected/2d4107db-fc48-4469-9991-8fade0c98697-kube-api-access-rck8s\") pod \"cert-manager-55b4d954d5-9pm9p\" (UID: \"2d4107db-fc48-4469-9991-8fade0c98697\") " pod="cert-manager/cert-manager-55b4d954d5-9pm9p"
Jul 30 00:23:11.614490 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:11.614455    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d4107db-fc48-4469-9991-8fade0c98697-bound-sa-token\") pod \"cert-manager-55b4d954d5-9pm9p\" (UID: \"2d4107db-fc48-4469-9991-8fade0c98697\") " pod="cert-manager/cert-manager-55b4d954d5-9pm9p"
Jul 30 00:23:11.614625 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:11.614532    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rck8s\" (UniqueName: \"kubernetes.io/projected/2d4107db-fc48-4469-9991-8fade0c98697-kube-api-access-rck8s\") pod \"cert-manager-55b4d954d5-9pm9p\" (UID: \"2d4107db-fc48-4469-9991-8fade0c98697\") " pod="cert-manager/cert-manager-55b4d954d5-9pm9p"
Jul 30 00:23:11.710209 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:11.710132    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9pm9p"
Jul 30 00:23:11.837487 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:11.837453    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9pm9p"]
Jul 30 00:23:11.840366 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:23:11.840330    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d4107db_fc48_4469_9991_8fade0c98697.slice/crio-a42f3be49f1085ceb25b167310cc820cc9dccd153100379c587445ab109d0b03 WatchSource:0}: Error finding container a42f3be49f1085ceb25b167310cc820cc9dccd153100379c587445ab109d0b03: Status 404 returned error can't find the container with id a42f3be49f1085ceb25b167310cc820cc9dccd153100379c587445ab109d0b03
Jul 30 00:23:11.842172 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:11.842156    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 00:23:12.698657 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:12.698620    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9pm9p" event={"ID":"2d4107db-fc48-4469-9991-8fade0c98697","Type":"ContainerStarted","Data":"a42f3be49f1085ceb25b167310cc820cc9dccd153100379c587445ab109d0b03"}
Jul 30 00:23:14.362625 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:14.362588    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-knr7k"]
Jul 30 00:23:14.366712 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:14.366685    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-knr7k"
Jul 30 00:23:14.369427 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:14.369376    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 30 00:23:14.369594 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:14.369454    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 30 00:23:14.370905 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:14.370881    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-jjjsf\""
Jul 30 00:23:14.372911 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:14.372887    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-knr7k"]
Jul 30 00:23:14.532750 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:14.532711    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96klb\" (UniqueName: \"kubernetes.io/projected/a21fa613-f28c-489c-89cd-337991e65557-kube-api-access-96klb\") pod \"openshift-lws-operator-fd8ccff4c-knr7k\" (UID: \"a21fa613-f28c-489c-89cd-337991e65557\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-knr7k"
Jul 30 00:23:14.532953 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:14.532854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a21fa613-f28c-489c-89cd-337991e65557-tmp\") pod \"openshift-lws-operator-fd8ccff4c-knr7k\" (UID: \"a21fa613-f28c-489c-89cd-337991e65557\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-knr7k"
Jul 30 00:23:14.634142 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:14.634101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a21fa613-f28c-489c-89cd-337991e65557-tmp\") pod \"openshift-lws-operator-fd8ccff4c-knr7k\" (UID: \"a21fa613-f28c-489c-89cd-337991e65557\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-knr7k"
Jul 30 00:23:14.634371 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:14.634206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96klb\" (UniqueName: \"kubernetes.io/projected/a21fa613-f28c-489c-89cd-337991e65557-kube-api-access-96klb\") pod \"openshift-lws-operator-fd8ccff4c-knr7k\" (UID: \"a21fa613-f28c-489c-89cd-337991e65557\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-knr7k"
Jul 30 00:23:14.634587 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:14.634556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a21fa613-f28c-489c-89cd-337991e65557-tmp\") pod \"openshift-lws-operator-fd8ccff4c-knr7k\" (UID: \"a21fa613-f28c-489c-89cd-337991e65557\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-knr7k"
Jul 30 00:23:14.643416 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:14.643388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96klb\" (UniqueName: \"kubernetes.io/projected/a21fa613-f28c-489c-89cd-337991e65557-kube-api-access-96klb\") pod \"openshift-lws-operator-fd8ccff4c-knr7k\" (UID: \"a21fa613-f28c-489c-89cd-337991e65557\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-knr7k"
Jul 30 00:23:14.679680 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:14.679649    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-knr7k"
Jul 30 00:23:16.022948 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:16.022890    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-knr7k"]
Jul 30 00:23:16.026479 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:23:16.026452    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda21fa613_f28c_489c_89cd_337991e65557.slice/crio-67adcf817ae6f04d8449bbab84f163295555b7b03ea312680c809ba69165b009 WatchSource:0}: Error finding container 67adcf817ae6f04d8449bbab84f163295555b7b03ea312680c809ba69165b009: Status 404 returned error can't find the container with id 67adcf817ae6f04d8449bbab84f163295555b7b03ea312680c809ba69165b009
Jul 30 00:23:16.718669 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:16.718627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9pm9p" event={"ID":"2d4107db-fc48-4469-9991-8fade0c98697","Type":"ContainerStarted","Data":"fa40e453d238ed655034b33bc703c52a621ca7b630b4639aa92a0232390da31a"}
Jul 30 00:23:16.719955 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:16.719929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-knr7k" event={"ID":"a21fa613-f28c-489c-89cd-337991e65557","Type":"ContainerStarted","Data":"67adcf817ae6f04d8449bbab84f163295555b7b03ea312680c809ba69165b009"}
Jul 30 00:23:16.734430 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:16.734388    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-9pm9p" podStartSLOduration=1.6117544449999999 podStartE2EDuration="5.734372528s" totalImagesPullingTime="4.122618083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:23:16.734018672 +0000 UTC m=+322.031285606" watchObservedRunningTime="2026-07-30 00:23:16.734372528 +0000 UTC m=+322.031639459"
Jul 30 00:23:19.733353 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:19.733318    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-knr7k" event={"ID":"a21fa613-f28c-489c-89cd-337991e65557","Type":"ContainerStarted","Data":"6299b02c6664f6966baf9c810c91ddb92075f8bb702128ea997efbc68df72277"}
Jul 30 00:23:19.750811 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:19.750754    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-knr7k" podStartSLOduration=3.0704930089999998 podStartE2EDuration="5.750738797s" totalImagesPullingTime="2.680245788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:23:19.748979432 +0000 UTC m=+325.046246363" watchObservedRunningTime="2026-07-30 00:23:19.750738797 +0000 UTC m=+325.048005728"
Jul 30 00:23:36.521414 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.521378    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l"]
Jul 30 00:23:36.524633 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.524611    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l"
Jul 30 00:23:36.528819 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.528792    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 30 00:23:36.528951 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.528792    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tjgkc\""
Jul 30 00:23:36.528951 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.528909    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 30 00:23:36.528951 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.528800    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 30 00:23:36.539729 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.539709    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l"]
Jul 30 00:23:36.614562 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.614531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/178cfbc9-70ed-48ac-ad08-c04cabcb55db-manager-config\") pod \"lws-controller-manager-685c7855bf-g9j9l\" (UID: \"178cfbc9-70ed-48ac-ad08-c04cabcb55db\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l"
Jul 30 00:23:36.614562 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.614559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/178cfbc9-70ed-48ac-ad08-c04cabcb55db-metrics-cert\") pod \"lws-controller-manager-685c7855bf-g9j9l\" (UID: \"178cfbc9-70ed-48ac-ad08-c04cabcb55db\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l"
Jul 30 00:23:36.614729 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.614659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jb89\" (UniqueName: \"kubernetes.io/projected/178cfbc9-70ed-48ac-ad08-c04cabcb55db-kube-api-access-8jb89\") pod \"lws-controller-manager-685c7855bf-g9j9l\" (UID: \"178cfbc9-70ed-48ac-ad08-c04cabcb55db\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l"
Jul 30 00:23:36.614729 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.614679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/178cfbc9-70ed-48ac-ad08-c04cabcb55db-cert\") pod \"lws-controller-manager-685c7855bf-g9j9l\" (UID: \"178cfbc9-70ed-48ac-ad08-c04cabcb55db\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l"
Jul 30 00:23:36.715192 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.715161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jb89\" (UniqueName: \"kubernetes.io/projected/178cfbc9-70ed-48ac-ad08-c04cabcb55db-kube-api-access-8jb89\") pod \"lws-controller-manager-685c7855bf-g9j9l\" (UID: \"178cfbc9-70ed-48ac-ad08-c04cabcb55db\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l"
Jul 30 00:23:36.715333 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.715196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/178cfbc9-70ed-48ac-ad08-c04cabcb55db-cert\") pod \"lws-controller-manager-685c7855bf-g9j9l\" (UID: \"178cfbc9-70ed-48ac-ad08-c04cabcb55db\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l"
Jul 30 00:23:36.715333 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.715224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/178cfbc9-70ed-48ac-ad08-c04cabcb55db-manager-config\") pod \"lws-controller-manager-685c7855bf-g9j9l\" (UID: \"178cfbc9-70ed-48ac-ad08-c04cabcb55db\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l"
Jul 30 00:23:36.715333 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.715289    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/178cfbc9-70ed-48ac-ad08-c04cabcb55db-metrics-cert\") pod \"lws-controller-manager-685c7855bf-g9j9l\" (UID: \"178cfbc9-70ed-48ac-ad08-c04cabcb55db\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l"
Jul 30 00:23:36.715975 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.715952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/178cfbc9-70ed-48ac-ad08-c04cabcb55db-manager-config\") pod \"lws-controller-manager-685c7855bf-g9j9l\" (UID: \"178cfbc9-70ed-48ac-ad08-c04cabcb55db\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l"
Jul 30 00:23:36.717725 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.717700    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/178cfbc9-70ed-48ac-ad08-c04cabcb55db-metrics-cert\") pod \"lws-controller-manager-685c7855bf-g9j9l\" (UID: \"178cfbc9-70ed-48ac-ad08-c04cabcb55db\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l"
Jul 30 00:23:36.717807 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.717777    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/178cfbc9-70ed-48ac-ad08-c04cabcb55db-cert\") pod \"lws-controller-manager-685c7855bf-g9j9l\" (UID: \"178cfbc9-70ed-48ac-ad08-c04cabcb55db\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l"
Jul 30 00:23:36.723756 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.723725    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jb89\" (UniqueName: \"kubernetes.io/projected/178cfbc9-70ed-48ac-ad08-c04cabcb55db-kube-api-access-8jb89\") pod \"lws-controller-manager-685c7855bf-g9j9l\" (UID: \"178cfbc9-70ed-48ac-ad08-c04cabcb55db\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l"
Jul 30 00:23:36.833869 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.833769    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l"
Jul 30 00:23:36.960522 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:36.960484    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l"]
Jul 30 00:23:36.963368 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:23:36.963337    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178cfbc9_70ed_48ac_ad08_c04cabcb55db.slice/crio-0f530993764d4a2511df4a0d4739f3fd74c3413fab72f13634e597df31519771 WatchSource:0}: Error finding container 0f530993764d4a2511df4a0d4739f3fd74c3413fab72f13634e597df31519771: Status 404 returned error can't find the container with id 0f530993764d4a2511df4a0d4739f3fd74c3413fab72f13634e597df31519771
Jul 30 00:23:37.803443 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:37.803405    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l" event={"ID":"178cfbc9-70ed-48ac-ad08-c04cabcb55db","Type":"ContainerStarted","Data":"0f530993764d4a2511df4a0d4739f3fd74c3413fab72f13634e597df31519771"}
Jul 30 00:23:38.998720 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:38.998691    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"]
Jul 30 00:23:39.003329 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.003309    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"
Jul 30 00:23:39.006822 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.006798    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 30 00:23:39.007429 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.007414    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 30 00:23:39.008493 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.008478    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 30 00:23:39.008546 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.008504    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kwj6t\""
Jul 30 00:23:39.008546 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.008536    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 30 00:23:39.027557 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.027531    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"]
Jul 30 00:23:39.136586 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.136561    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/27f4f29e-3606-46f6-aaa6-6bae1807ace8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b74bc5486-z878b\" (UID: \"27f4f29e-3606-46f6-aaa6-6bae1807ace8\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"
Jul 30 00:23:39.136723 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.136614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnqv9\" (UniqueName: \"kubernetes.io/projected/27f4f29e-3606-46f6-aaa6-6bae1807ace8-kube-api-access-vnqv9\") pod \"opendatahub-operator-controller-manager-5b74bc5486-z878b\" (UID: \"27f4f29e-3606-46f6-aaa6-6bae1807ace8\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"
Jul 30 00:23:39.136723 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.136684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/27f4f29e-3606-46f6-aaa6-6bae1807ace8-manifests\") pod \"opendatahub-operator-controller-manager-5b74bc5486-z878b\" (UID: \"27f4f29e-3606-46f6-aaa6-6bae1807ace8\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"
Jul 30 00:23:39.136818 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.136725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27f4f29e-3606-46f6-aaa6-6bae1807ace8-tmp\") pod \"opendatahub-operator-controller-manager-5b74bc5486-z878b\" (UID: \"27f4f29e-3606-46f6-aaa6-6bae1807ace8\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"
Jul 30 00:23:39.136818 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.136783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27f4f29e-3606-46f6-aaa6-6bae1807ace8-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b74bc5486-z878b\" (UID: \"27f4f29e-3606-46f6-aaa6-6bae1807ace8\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"
Jul 30 00:23:39.237950 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.237920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27f4f29e-3606-46f6-aaa6-6bae1807ace8-tmp\") pod \"opendatahub-operator-controller-manager-5b74bc5486-z878b\" (UID: \"27f4f29e-3606-46f6-aaa6-6bae1807ace8\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"
Jul 30 00:23:39.238093 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.237962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27f4f29e-3606-46f6-aaa6-6bae1807ace8-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b74bc5486-z878b\" (UID: \"27f4f29e-3606-46f6-aaa6-6bae1807ace8\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"
Jul 30 00:23:39.238093 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.237990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/27f4f29e-3606-46f6-aaa6-6bae1807ace8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b74bc5486-z878b\" (UID: \"27f4f29e-3606-46f6-aaa6-6bae1807ace8\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"
Jul 30 00:23:39.238205 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.238182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnqv9\" (UniqueName: \"kubernetes.io/projected/27f4f29e-3606-46f6-aaa6-6bae1807ace8-kube-api-access-vnqv9\") pod \"opendatahub-operator-controller-manager-5b74bc5486-z878b\" (UID: \"27f4f29e-3606-46f6-aaa6-6bae1807ace8\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"
Jul 30 00:23:39.238301 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.238281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/27f4f29e-3606-46f6-aaa6-6bae1807ace8-manifests\") pod \"opendatahub-operator-controller-manager-5b74bc5486-z878b\" (UID: \"27f4f29e-3606-46f6-aaa6-6bae1807ace8\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"
Jul 30 00:23:39.238687 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.238661    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/27f4f29e-3606-46f6-aaa6-6bae1807ace8-manifests\") pod \"opendatahub-operator-controller-manager-5b74bc5486-z878b\" (UID: \"27f4f29e-3606-46f6-aaa6-6bae1807ace8\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"
Jul 30 00:23:39.240344 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.240316    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27f4f29e-3606-46f6-aaa6-6bae1807ace8-tmp\") pod \"opendatahub-operator-controller-manager-5b74bc5486-z878b\" (UID: \"27f4f29e-3606-46f6-aaa6-6bae1807ace8\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"
Jul 30 00:23:39.240432 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.240416    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/27f4f29e-3606-46f6-aaa6-6bae1807ace8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b74bc5486-z878b\" (UID: \"27f4f29e-3606-46f6-aaa6-6bae1807ace8\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"
Jul 30 00:23:39.240520 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.240503    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27f4f29e-3606-46f6-aaa6-6bae1807ace8-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b74bc5486-z878b\" (UID: \"27f4f29e-3606-46f6-aaa6-6bae1807ace8\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"
Jul 30 00:23:39.247093 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.247072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnqv9\" (UniqueName: \"kubernetes.io/projected/27f4f29e-3606-46f6-aaa6-6bae1807ace8-kube-api-access-vnqv9\") pod \"opendatahub-operator-controller-manager-5b74bc5486-z878b\" (UID: \"27f4f29e-3606-46f6-aaa6-6bae1807ace8\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"
Jul 30 00:23:39.315162 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.315076    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"
Jul 30 00:23:39.468203 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.468169    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"]
Jul 30 00:23:39.472340 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:23:39.472309    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f4f29e_3606_46f6_aaa6_6bae1807ace8.slice/crio-2271e0bea69ecbe8271ea02a2d74ba34e08c340088d4ed76ff55c0b6a4f19eb4 WatchSource:0}: Error finding container 2271e0bea69ecbe8271ea02a2d74ba34e08c340088d4ed76ff55c0b6a4f19eb4: Status 404 returned error can't find the container with id 2271e0bea69ecbe8271ea02a2d74ba34e08c340088d4ed76ff55c0b6a4f19eb4
Jul 30 00:23:39.813817 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:39.813783    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b" event={"ID":"27f4f29e-3606-46f6-aaa6-6bae1807ace8","Type":"ContainerStarted","Data":"2271e0bea69ecbe8271ea02a2d74ba34e08c340088d4ed76ff55c0b6a4f19eb4"}
Jul 30 00:23:40.822300 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:40.822258    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l" event={"ID":"178cfbc9-70ed-48ac-ad08-c04cabcb55db","Type":"ContainerStarted","Data":"176213138aaf07c164ab14a3a50214e8fa53c3c5a6701cd0e1519ac8b29c74ff"}
Jul 30 00:23:40.822804 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:40.822500    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l"
Jul 30 00:23:40.848690 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:40.848631    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l" podStartSLOduration=1.620442765 podStartE2EDuration="4.848609979s" totalImagesPullingTime="3.228167214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:23:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:23:40.846670956 +0000 UTC m=+346.143937888" watchObservedRunningTime="2026-07-30 00:23:40.848609979 +0000 UTC m=+346.145876913"
Jul 30 00:23:42.833234 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:42.833195    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b" event={"ID":"27f4f29e-3606-46f6-aaa6-6bae1807ace8","Type":"ContainerStarted","Data":"0744f8a0e19dd132bfd6b08e67f53a0d9abaf664addb82967175c86207fc77a7"}
Jul 30 00:23:43.839858 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:43.839826    2579 generic.go:356] "Generic (PLEG): container finished" podID="27f4f29e-3606-46f6-aaa6-6bae1807ace8" containerID="0744f8a0e19dd132bfd6b08e67f53a0d9abaf664addb82967175c86207fc77a7" exitCode=0
Jul 30 00:23:43.840271 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:43.839897    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b" event={"ID":"27f4f29e-3606-46f6-aaa6-6bae1807ace8","Type":"ContainerDied","Data":"0744f8a0e19dd132bfd6b08e67f53a0d9abaf664addb82967175c86207fc77a7"}
Jul 30 00:23:44.846762 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:44.846657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b" event={"ID":"27f4f29e-3606-46f6-aaa6-6bae1807ace8","Type":"ContainerStarted","Data":"9318ee4264ddd69af96e7957f265f3b834d3cf0ee6218cd03dde18d44035b485"}
Jul 30 00:23:44.847184 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:44.846778    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"
Jul 30 00:23:44.872581 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:44.872522    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b" podStartSLOduration=2.129768329 podStartE2EDuration="6.872500979s" totalImagesPullingTime="3.692124369s" totalInitContainerRuntime="1.050608281s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:23:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:23:44.870650185 +0000 UTC m=+350.167917118" watchObservedRunningTime="2026-07-30 00:23:44.872500979 +0000 UTC m=+350.169767909"
Jul 30 00:23:51.830021 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:51.829939    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-685c7855bf-g9j9l"
Jul 30 00:23:55.852732 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:55.852698    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-z878b"
Jul 30 00:23:59.189916 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.189878    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-94d649dc8-lm6p2"]
Jul 30 00:23:59.200783 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.200756    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-94d649dc8-lm6p2"
Jul 30 00:23:59.203296 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.203267    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-94d649dc8-lm6p2"]
Jul 30 00:23:59.205164 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.205140    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wvrtc\""
Jul 30 00:23:59.205676 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.205496    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 30 00:23:59.205808 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.205776    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 30 00:23:59.206365 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.206341    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 30 00:23:59.206729 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.206712    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 30 00:23:59.313763 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.313727    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6c5p\" (UniqueName: \"kubernetes.io/projected/fb873962-c767-48e6-b6b3-09435499efa5-kube-api-access-w6c5p\") pod \"kube-auth-proxy-94d649dc8-lm6p2\" (UID: \"fb873962-c767-48e6-b6b3-09435499efa5\") " pod="openshift-ingress/kube-auth-proxy-94d649dc8-lm6p2"
Jul 30 00:23:59.313942 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.313784    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb873962-c767-48e6-b6b3-09435499efa5-tmp\") pod \"kube-auth-proxy-94d649dc8-lm6p2\" (UID: \"fb873962-c767-48e6-b6b3-09435499efa5\") " pod="openshift-ingress/kube-auth-proxy-94d649dc8-lm6p2"
Jul 30 00:23:59.313942 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.313899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb873962-c767-48e6-b6b3-09435499efa5-tls-certs\") pod \"kube-auth-proxy-94d649dc8-lm6p2\" (UID: \"fb873962-c767-48e6-b6b3-09435499efa5\") " pod="openshift-ingress/kube-auth-proxy-94d649dc8-lm6p2"
Jul 30 00:23:59.414995 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.414960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb873962-c767-48e6-b6b3-09435499efa5-tls-certs\") pod \"kube-auth-proxy-94d649dc8-lm6p2\" (UID: \"fb873962-c767-48e6-b6b3-09435499efa5\") " pod="openshift-ingress/kube-auth-proxy-94d649dc8-lm6p2"
Jul 30 00:23:59.415160 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.415023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6c5p\" (UniqueName: \"kubernetes.io/projected/fb873962-c767-48e6-b6b3-09435499efa5-kube-api-access-w6c5p\") pod \"kube-auth-proxy-94d649dc8-lm6p2\" (UID: \"fb873962-c767-48e6-b6b3-09435499efa5\") " pod="openshift-ingress/kube-auth-proxy-94d649dc8-lm6p2"
Jul 30 00:23:59.415160 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.415087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb873962-c767-48e6-b6b3-09435499efa5-tmp\") pod \"kube-auth-proxy-94d649dc8-lm6p2\" (UID: \"fb873962-c767-48e6-b6b3-09435499efa5\") " pod="openshift-ingress/kube-auth-proxy-94d649dc8-lm6p2"
Jul 30 00:23:59.417400 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.417369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb873962-c767-48e6-b6b3-09435499efa5-tmp\") pod \"kube-auth-proxy-94d649dc8-lm6p2\" (UID: \"fb873962-c767-48e6-b6b3-09435499efa5\") " pod="openshift-ingress/kube-auth-proxy-94d649dc8-lm6p2"
Jul 30 00:23:59.418162 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.418142    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb873962-c767-48e6-b6b3-09435499efa5-tls-certs\") pod \"kube-auth-proxy-94d649dc8-lm6p2\" (UID: \"fb873962-c767-48e6-b6b3-09435499efa5\") " pod="openshift-ingress/kube-auth-proxy-94d649dc8-lm6p2"
Jul 30 00:23:59.423148 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.423124    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6c5p\" (UniqueName: \"kubernetes.io/projected/fb873962-c767-48e6-b6b3-09435499efa5-kube-api-access-w6c5p\") pod \"kube-auth-proxy-94d649dc8-lm6p2\" (UID: \"fb873962-c767-48e6-b6b3-09435499efa5\") " pod="openshift-ingress/kube-auth-proxy-94d649dc8-lm6p2"
Jul 30 00:23:59.516110 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.516012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-94d649dc8-lm6p2"
Jul 30 00:23:59.677431 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.677333    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-94d649dc8-lm6p2"]
Jul 30 00:23:59.680283 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:23:59.680231    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb873962_c767_48e6_b6b3_09435499efa5.slice/crio-06c7f19b60030bbfc91498dd02a3582d890b0bc78e45d12a4b10d9980c2314df WatchSource:0}: Error finding container 06c7f19b60030bbfc91498dd02a3582d890b0bc78e45d12a4b10d9980c2314df: Status 404 returned error can't find the container with id 06c7f19b60030bbfc91498dd02a3582d890b0bc78e45d12a4b10d9980c2314df
Jul 30 00:23:59.914293 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:23:59.914236    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-94d649dc8-lm6p2" event={"ID":"fb873962-c767-48e6-b6b3-09435499efa5","Type":"ContainerStarted","Data":"06c7f19b60030bbfc91498dd02a3582d890b0bc78e45d12a4b10d9980c2314df"}
Jul 30 00:24:03.934293 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:24:03.934234    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-94d649dc8-lm6p2" event={"ID":"fb873962-c767-48e6-b6b3-09435499efa5","Type":"ContainerStarted","Data":"0f52a857f5bb47a077b4aa0bfc5d9a217bde09c03ca5a4b46a9b9f738ca13313"}
Jul 30 00:24:03.952946 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:24:03.952880    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-94d649dc8-lm6p2" podStartSLOduration=1.6943872469999999 podStartE2EDuration="4.952858419s" totalImagesPullingTime="3.258471172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:23:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:24:03.950261359 +0000 UTC m=+369.247528285" watchObservedRunningTime="2026-07-30 00:24:03.952858419 +0000 UTC m=+369.250125353"
Jul 30 00:25:25.519300 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.519200    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b5764f55-52b46"]
Jul 30 00:25:25.521398 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.521380    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.525330 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.525295    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pfswq\""
Jul 30 00:25:25.525487 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.525358    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 30 00:25:25.525487 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.525433    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 30 00:25:25.525487 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.525455    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 30 00:25:25.525648 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.525489    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 30 00:25:25.525648 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.525499    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 30 00:25:25.525648 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.525511    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 30 00:25:25.526922 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.526908    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 30 00:25:25.531018 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.530998    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 30 00:25:25.532496 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.532474    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5764f55-52b46"]
Jul 30 00:25:25.651830 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.651785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b64488b7-69fd-400d-8028-7d6df50c4d45-console-serving-cert\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.652024 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.651856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b64488b7-69fd-400d-8028-7d6df50c4d45-console-oauth-config\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.652024 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.651977    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b64488b7-69fd-400d-8028-7d6df50c4d45-trusted-ca-bundle\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.652024 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.652009    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnrpw\" (UniqueName: \"kubernetes.io/projected/b64488b7-69fd-400d-8028-7d6df50c4d45-kube-api-access-tnrpw\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.652144 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.652026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b64488b7-69fd-400d-8028-7d6df50c4d45-oauth-serving-cert\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.652144 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.652110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b64488b7-69fd-400d-8028-7d6df50c4d45-console-config\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.652144 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.652141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b64488b7-69fd-400d-8028-7d6df50c4d45-service-ca\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.753650 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.753610    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b64488b7-69fd-400d-8028-7d6df50c4d45-trusted-ca-bundle\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.753650 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.753646    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnrpw\" (UniqueName: \"kubernetes.io/projected/b64488b7-69fd-400d-8028-7d6df50c4d45-kube-api-access-tnrpw\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.753650 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.753664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b64488b7-69fd-400d-8028-7d6df50c4d45-oauth-serving-cert\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.753968 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.753694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b64488b7-69fd-400d-8028-7d6df50c4d45-console-config\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.753968 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.753720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b64488b7-69fd-400d-8028-7d6df50c4d45-service-ca\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.753968 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.753747    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b64488b7-69fd-400d-8028-7d6df50c4d45-console-serving-cert\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.753968 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.753779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b64488b7-69fd-400d-8028-7d6df50c4d45-console-oauth-config\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.754546 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.754522    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b64488b7-69fd-400d-8028-7d6df50c4d45-oauth-serving-cert\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.754678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.754544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b64488b7-69fd-400d-8028-7d6df50c4d45-console-config\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.754678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.754544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b64488b7-69fd-400d-8028-7d6df50c4d45-service-ca\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.754954 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.754931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b64488b7-69fd-400d-8028-7d6df50c4d45-trusted-ca-bundle\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.756482 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.756457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b64488b7-69fd-400d-8028-7d6df50c4d45-console-oauth-config\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.756578 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.756499    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b64488b7-69fd-400d-8028-7d6df50c4d45-console-serving-cert\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.761624 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.761602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnrpw\" (UniqueName: \"kubernetes.io/projected/b64488b7-69fd-400d-8028-7d6df50c4d45-kube-api-access-tnrpw\") pod \"console-b5764f55-52b46\" (UID: \"b64488b7-69fd-400d-8028-7d6df50c4d45\") " pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.832837 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.832757    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:25.966893 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:25.966862    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5764f55-52b46"]
Jul 30 00:25:25.970461 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:25:25.970432    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64488b7_69fd_400d_8028_7d6df50c4d45.slice/crio-ae459d49333d45ab5381dc6990dbd09c5abc36bc8d12ab052ee4ec1a3fa3eb28 WatchSource:0}: Error finding container ae459d49333d45ab5381dc6990dbd09c5abc36bc8d12ab052ee4ec1a3fa3eb28: Status 404 returned error can't find the container with id ae459d49333d45ab5381dc6990dbd09c5abc36bc8d12ab052ee4ec1a3fa3eb28
Jul 30 00:25:26.290885 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:26.290848    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5764f55-52b46" event={"ID":"b64488b7-69fd-400d-8028-7d6df50c4d45","Type":"ContainerStarted","Data":"a8624704ec0a9dac4e0ae3345c1d6751c39991e4642dc495fcbe532541b918c2"}
Jul 30 00:25:26.291110 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:26.290899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5764f55-52b46" event={"ID":"b64488b7-69fd-400d-8028-7d6df50c4d45","Type":"ContainerStarted","Data":"ae459d49333d45ab5381dc6990dbd09c5abc36bc8d12ab052ee4ec1a3fa3eb28"}
Jul 30 00:25:26.308331 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:26.308282    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b5764f55-52b46" podStartSLOduration=1.308255557 podStartE2EDuration="1.308255557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:25:26.306993006 +0000 UTC m=+451.604259953" watchObservedRunningTime="2026-07-30 00:25:26.308255557 +0000 UTC m=+451.605522479"
Jul 30 00:25:35.833172 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:35.833137    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:35.833738 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:35.833278    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:35.837859 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:35.837837    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:36.338236 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:36.338211    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b5764f55-52b46"
Jul 30 00:25:49.618463 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:49.618424    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-h8mmx"]
Jul 30 00:25:49.620642 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:49.620626    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h8mmx"
Jul 30 00:25:49.623471 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:49.623435    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 30 00:25:49.623471 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:49.623440    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 30 00:25:49.623655 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:49.623480    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 30 00:25:49.624817 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:49.624801    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4v4jb\""
Jul 30 00:25:49.624914 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:49.624804    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 30 00:25:49.630884 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:49.630858    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-h8mmx"]
Jul 30 00:25:49.749665 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:49.749638    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fbd644a-580f-44e0-89cf-121ac1eb0c80-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-h8mmx\" (UID: \"7fbd644a-580f-44e0-89cf-121ac1eb0c80\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h8mmx"
Jul 30 00:25:49.749803 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:49.749716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7fbd644a-580f-44e0-89cf-121ac1eb0c80-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-h8mmx\" (UID: \"7fbd644a-580f-44e0-89cf-121ac1eb0c80\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h8mmx"
Jul 30 00:25:49.749803 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:49.749740    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb54z\" (UniqueName: \"kubernetes.io/projected/7fbd644a-580f-44e0-89cf-121ac1eb0c80-kube-api-access-jb54z\") pod \"kuadrant-console-plugin-66b5774985-h8mmx\" (UID: \"7fbd644a-580f-44e0-89cf-121ac1eb0c80\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h8mmx"
Jul 30 00:25:49.850137 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:49.850106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7fbd644a-580f-44e0-89cf-121ac1eb0c80-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-h8mmx\" (UID: \"7fbd644a-580f-44e0-89cf-121ac1eb0c80\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h8mmx"
Jul 30 00:25:49.850317 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:49.850141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb54z\" (UniqueName: \"kubernetes.io/projected/7fbd644a-580f-44e0-89cf-121ac1eb0c80-kube-api-access-jb54z\") pod \"kuadrant-console-plugin-66b5774985-h8mmx\" (UID: \"7fbd644a-580f-44e0-89cf-121ac1eb0c80\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h8mmx"
Jul 30 00:25:49.850317 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:49.850280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fbd644a-580f-44e0-89cf-121ac1eb0c80-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-h8mmx\" (UID: \"7fbd644a-580f-44e0-89cf-121ac1eb0c80\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h8mmx"
Jul 30 00:25:49.850836 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:49.850803    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7fbd644a-580f-44e0-89cf-121ac1eb0c80-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-h8mmx\" (UID: \"7fbd644a-580f-44e0-89cf-121ac1eb0c80\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h8mmx"
Jul 30 00:25:49.852895 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:49.852875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fbd644a-580f-44e0-89cf-121ac1eb0c80-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-h8mmx\" (UID: \"7fbd644a-580f-44e0-89cf-121ac1eb0c80\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h8mmx"
Jul 30 00:25:49.857875 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:49.857848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb54z\" (UniqueName: \"kubernetes.io/projected/7fbd644a-580f-44e0-89cf-121ac1eb0c80-kube-api-access-jb54z\") pod \"kuadrant-console-plugin-66b5774985-h8mmx\" (UID: \"7fbd644a-580f-44e0-89cf-121ac1eb0c80\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h8mmx"
Jul 30 00:25:49.931486 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:49.931463    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h8mmx"
Jul 30 00:25:50.060753 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:50.060728    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-h8mmx"]
Jul 30 00:25:50.063460 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:25:50.063435    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fbd644a_580f_44e0_89cf_121ac1eb0c80.slice/crio-9388f8b701d3d31d4a70950d93051a0c2a976b048aa81ee51fadbda58350d4cf WatchSource:0}: Error finding container 9388f8b701d3d31d4a70950d93051a0c2a976b048aa81ee51fadbda58350d4cf: Status 404 returned error can't find the container with id 9388f8b701d3d31d4a70950d93051a0c2a976b048aa81ee51fadbda58350d4cf
Jul 30 00:25:50.396585 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:50.396553    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h8mmx" event={"ID":"7fbd644a-580f-44e0-89cf-121ac1eb0c80","Type":"ContainerStarted","Data":"9388f8b701d3d31d4a70950d93051a0c2a976b048aa81ee51fadbda58350d4cf"}
Jul 30 00:25:59.997859 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:25:59.997818    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"]
Jul 30 00:26:00.004973 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.004944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"
Jul 30 00:26:00.008431 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.008407    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hmwnt\""
Jul 30 00:26:00.019613 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.019584    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"]
Jul 30 00:26:00.152583 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.152547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnl92\" (UniqueName: \"kubernetes.io/projected/9e4cdd85-530f-4a67-a488-379e4ae2340b-kube-api-access-gnl92\") pod \"kuadrant-operator-controller-manager-55467d9f7-j5zjc\" (UID: \"9e4cdd85-530f-4a67-a488-379e4ae2340b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"
Jul 30 00:26:00.152784 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.152635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e4cdd85-530f-4a67-a488-379e4ae2340b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-j5zjc\" (UID: \"9e4cdd85-530f-4a67-a488-379e4ae2340b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"
Jul 30 00:26:00.253914 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.253824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnl92\" (UniqueName: \"kubernetes.io/projected/9e4cdd85-530f-4a67-a488-379e4ae2340b-kube-api-access-gnl92\") pod \"kuadrant-operator-controller-manager-55467d9f7-j5zjc\" (UID: \"9e4cdd85-530f-4a67-a488-379e4ae2340b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"
Jul 30 00:26:00.253914 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.253896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e4cdd85-530f-4a67-a488-379e4ae2340b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-j5zjc\" (UID: \"9e4cdd85-530f-4a67-a488-379e4ae2340b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"
Jul 30 00:26:00.254403 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.254380    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e4cdd85-530f-4a67-a488-379e4ae2340b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-j5zjc\" (UID: \"9e4cdd85-530f-4a67-a488-379e4ae2340b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"
Jul 30 00:26:00.268165 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.268137    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnl92\" (UniqueName: \"kubernetes.io/projected/9e4cdd85-530f-4a67-a488-379e4ae2340b-kube-api-access-gnl92\") pod \"kuadrant-operator-controller-manager-55467d9f7-j5zjc\" (UID: \"9e4cdd85-530f-4a67-a488-379e4ae2340b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"
Jul 30 00:26:00.318504 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.318471    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"
Jul 30 00:26:00.530374 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.530279    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"]
Jul 30 00:26:00.534423 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.534392    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"]
Jul 30 00:26:00.560634 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.560605    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z"]
Jul 30 00:26:00.569087 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.569060    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z"
Jul 30 00:26:00.581806 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.581777    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z"]
Jul 30 00:26:00.656583 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.656546    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e56efbdc-400c-4671-bf3d-e08658f36c08-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-qxp7z\" (UID: \"e56efbdc-400c-4671-bf3d-e08658f36c08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z"
Jul 30 00:26:00.656746 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.656589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzqbs\" (UniqueName: \"kubernetes.io/projected/e56efbdc-400c-4671-bf3d-e08658f36c08-kube-api-access-gzqbs\") pod \"kuadrant-operator-controller-manager-55467d9f7-qxp7z\" (UID: \"e56efbdc-400c-4671-bf3d-e08658f36c08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z"
Jul 30 00:26:00.757829 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.757787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e56efbdc-400c-4671-bf3d-e08658f36c08-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-qxp7z\" (UID: \"e56efbdc-400c-4671-bf3d-e08658f36c08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z"
Jul 30 00:26:00.758027 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.757839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzqbs\" (UniqueName: \"kubernetes.io/projected/e56efbdc-400c-4671-bf3d-e08658f36c08-kube-api-access-gzqbs\") pod \"kuadrant-operator-controller-manager-55467d9f7-qxp7z\" (UID: \"e56efbdc-400c-4671-bf3d-e08658f36c08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z"
Jul 30 00:26:00.758378 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.758350    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e56efbdc-400c-4671-bf3d-e08658f36c08-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-qxp7z\" (UID: \"e56efbdc-400c-4671-bf3d-e08658f36c08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z"
Jul 30 00:26:00.777810 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.777778    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzqbs\" (UniqueName: \"kubernetes.io/projected/e56efbdc-400c-4671-bf3d-e08658f36c08-kube-api-access-gzqbs\") pod \"kuadrant-operator-controller-manager-55467d9f7-qxp7z\" (UID: \"e56efbdc-400c-4671-bf3d-e08658f36c08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z"
Jul 30 00:26:00.885865 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:00.885834    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z"
Jul 30 00:26:13.474077 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:13.474053    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z"]
Jul 30 00:26:13.476669 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:26:13.476640    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56efbdc_400c_4671_bf3d_e08658f36c08.slice/crio-b079e9f2657ed55834c513f08412f19d018c19a12d433de89d11265f998b2619 WatchSource:0}: Error finding container b079e9f2657ed55834c513f08412f19d018c19a12d433de89d11265f998b2619: Status 404 returned error can't find the container with id b079e9f2657ed55834c513f08412f19d018c19a12d433de89d11265f998b2619
Jul 30 00:26:13.519946 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:13.519913    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z" event={"ID":"e56efbdc-400c-4671-bf3d-e08658f36c08","Type":"ContainerStarted","Data":"b079e9f2657ed55834c513f08412f19d018c19a12d433de89d11265f998b2619"}
Jul 30 00:26:14.526554 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:14.526520    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h8mmx" event={"ID":"7fbd644a-580f-44e0-89cf-121ac1eb0c80","Type":"ContainerStarted","Data":"2b224914144c1b8a165d3b7b056e69994010c236f0b65c2f0cd422a7b26cd4d0"}
Jul 30 00:26:14.541628 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:14.541580    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h8mmx" podStartSLOduration=1.883795069 podStartE2EDuration="25.541563397s" totalImagesPullingTime="23.657768328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:26:14.541005487 +0000 UTC m=+499.838272420" watchObservedRunningTime="2026-07-30 00:26:14.541563397 +0000 UTC m=+499.838830382"
Jul 30 00:26:15.879693 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:26:15.879654    2579 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-j5zjc_kuadrant-system_9e4cdd85-530f-4a67-a488-379e4ae2340b_0(a9bc142119513c4b892c0af2874399a3a5eecba1a38cc6745498a0055b92c919): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-j5zjc to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}"
Jul 30 00:26:15.880041 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:26:15.879719    2579 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-j5zjc_kuadrant-system_9e4cdd85-530f-4a67-a488-379e4ae2340b_0(a9bc142119513c4b892c0af2874399a3a5eecba1a38cc6745498a0055b92c919): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-j5zjc to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"
Jul 30 00:26:16.535092 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:16.535062    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"
Jul 30 00:26:16.537910 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:26:16.537865    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-j5zjc\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="9e4cdd85-530f-4a67-a488-379e4ae2340b" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"
Jul 30 00:26:16.542771 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:16.542751    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"
Jul 30 00:26:16.545132 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:26:16.545100    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-j5zjc\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="9e4cdd85-530f-4a67-a488-379e4ae2340b" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"
Jul 30 00:26:16.611328 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:16.611297    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e4cdd85-530f-4a67-a488-379e4ae2340b-extensions-socket-volume\") pod \"9e4cdd85-530f-4a67-a488-379e4ae2340b\" (UID: \"9e4cdd85-530f-4a67-a488-379e4ae2340b\") "
Jul 30 00:26:16.611612 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:16.611584    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e4cdd85-530f-4a67-a488-379e4ae2340b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9e4cdd85-530f-4a67-a488-379e4ae2340b" (UID: "9e4cdd85-530f-4a67-a488-379e4ae2340b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:26:16.712705 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:16.712671    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnl92\" (UniqueName: \"kubernetes.io/projected/9e4cdd85-530f-4a67-a488-379e4ae2340b-kube-api-access-gnl92\") pod \"9e4cdd85-530f-4a67-a488-379e4ae2340b\" (UID: \"9e4cdd85-530f-4a67-a488-379e4ae2340b\") "
Jul 30 00:26:16.712928 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:16.712915    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e4cdd85-530f-4a67-a488-379e4ae2340b-extensions-socket-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:26:16.715144 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:16.715114    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4cdd85-530f-4a67-a488-379e4ae2340b-kube-api-access-gnl92" (OuterVolumeSpecName: "kube-api-access-gnl92") pod "9e4cdd85-530f-4a67-a488-379e4ae2340b" (UID: "9e4cdd85-530f-4a67-a488-379e4ae2340b"). InnerVolumeSpecName "kube-api-access-gnl92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:26:16.813907 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:16.813829    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnl92\" (UniqueName: \"kubernetes.io/projected/9e4cdd85-530f-4a67-a488-379e4ae2340b-kube-api-access-gnl92\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:26:17.325369 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:17.325322    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e4cdd85-530f-4a67-a488-379e4ae2340b" path="/var/lib/kubelet/pods/9e4cdd85-530f-4a67-a488-379e4ae2340b/volumes"
Jul 30 00:26:17.540282 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:17.540236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"
Jul 30 00:26:17.542987 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:26:17.542949    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-j5zjc\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="9e4cdd85-530f-4a67-a488-379e4ae2340b" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"
Jul 30 00:26:17.545092 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:26:17.545064    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-j5zjc\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="9e4cdd85-530f-4a67-a488-379e4ae2340b" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"
Jul 30 00:26:18.546780 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:18.546742    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z" event={"ID":"e56efbdc-400c-4671-bf3d-e08658f36c08","Type":"ContainerStarted","Data":"80b458e6c5ae22ef786fc5dfd40629fce29ea8d189c8d35055402256078d281a"}
Jul 30 00:26:18.547150 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:18.546846    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z"
Jul 30 00:26:18.549502 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:26:18.549472    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-j5zjc\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="9e4cdd85-530f-4a67-a488-379e4ae2340b" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-j5zjc"
Jul 30 00:26:18.602640 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:18.602549    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z" podStartSLOduration=13.743410163 podStartE2EDuration="18.602534573s" totalImagesPullingTime="4.85912441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:26:18.599815109 +0000 UTC m=+503.897082040" watchObservedRunningTime="2026-07-30 00:26:18.602534573 +0000 UTC m=+503.899801505"
Jul 30 00:26:29.553861 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:29.553832    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z"
Jul 30 00:26:42.067271 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:42.067224    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z"]
Jul 30 00:26:42.067739 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:42.067435    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z" podUID="e56efbdc-400c-4671-bf3d-e08658f36c08" containerName="manager" containerID="cri-o://80b458e6c5ae22ef786fc5dfd40629fce29ea8d189c8d35055402256078d281a" gracePeriod=10
Jul 30 00:26:42.319185 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:42.319119    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z"
Jul 30 00:26:42.422834 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:42.422803    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e56efbdc-400c-4671-bf3d-e08658f36c08-extensions-socket-volume\") pod \"e56efbdc-400c-4671-bf3d-e08658f36c08\" (UID: \"e56efbdc-400c-4671-bf3d-e08658f36c08\") "
Jul 30 00:26:42.423007 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:42.422869    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzqbs\" (UniqueName: \"kubernetes.io/projected/e56efbdc-400c-4671-bf3d-e08658f36c08-kube-api-access-gzqbs\") pod \"e56efbdc-400c-4671-bf3d-e08658f36c08\" (UID: \"e56efbdc-400c-4671-bf3d-e08658f36c08\") "
Jul 30 00:26:42.423331 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:42.423307    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56efbdc-400c-4671-bf3d-e08658f36c08-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e56efbdc-400c-4671-bf3d-e08658f36c08" (UID: "e56efbdc-400c-4671-bf3d-e08658f36c08"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:26:42.425074 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:42.425050    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56efbdc-400c-4671-bf3d-e08658f36c08-kube-api-access-gzqbs" (OuterVolumeSpecName: "kube-api-access-gzqbs") pod "e56efbdc-400c-4671-bf3d-e08658f36c08" (UID: "e56efbdc-400c-4671-bf3d-e08658f36c08"). InnerVolumeSpecName "kube-api-access-gzqbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:26:42.524219 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:42.524195    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e56efbdc-400c-4671-bf3d-e08658f36c08-extensions-socket-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:26:42.524219 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:42.524217    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzqbs\" (UniqueName: \"kubernetes.io/projected/e56efbdc-400c-4671-bf3d-e08658f36c08-kube-api-access-gzqbs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:26:42.656228 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:42.656200    2579 generic.go:356] "Generic (PLEG): container finished" podID="e56efbdc-400c-4671-bf3d-e08658f36c08" containerID="80b458e6c5ae22ef786fc5dfd40629fce29ea8d189c8d35055402256078d281a" exitCode=0
Jul 30 00:26:42.656362 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:42.656259    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z"
Jul 30 00:26:42.656362 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:42.656294    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z" event={"ID":"e56efbdc-400c-4671-bf3d-e08658f36c08","Type":"ContainerDied","Data":"80b458e6c5ae22ef786fc5dfd40629fce29ea8d189c8d35055402256078d281a"}
Jul 30 00:26:42.656362 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:42.656346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z" event={"ID":"e56efbdc-400c-4671-bf3d-e08658f36c08","Type":"ContainerDied","Data":"b079e9f2657ed55834c513f08412f19d018c19a12d433de89d11265f998b2619"}
Jul 30 00:26:42.656465 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:42.656371    2579 scope.go:117] "RemoveContainer" containerID="80b458e6c5ae22ef786fc5dfd40629fce29ea8d189c8d35055402256078d281a"
Jul 30 00:26:42.665380 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:42.665362    2579 scope.go:117] "RemoveContainer" containerID="80b458e6c5ae22ef786fc5dfd40629fce29ea8d189c8d35055402256078d281a"
Jul 30 00:26:42.665617 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:26:42.665601    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b458e6c5ae22ef786fc5dfd40629fce29ea8d189c8d35055402256078d281a\": container with ID starting with 80b458e6c5ae22ef786fc5dfd40629fce29ea8d189c8d35055402256078d281a not found: ID does not exist" containerID="80b458e6c5ae22ef786fc5dfd40629fce29ea8d189c8d35055402256078d281a"
Jul 30 00:26:42.665683 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:42.665626    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b458e6c5ae22ef786fc5dfd40629fce29ea8d189c8d35055402256078d281a"} err="failed to get container status \"80b458e6c5ae22ef786fc5dfd40629fce29ea8d189c8d35055402256078d281a\": rpc error: code = NotFound desc = could not find container \"80b458e6c5ae22ef786fc5dfd40629fce29ea8d189c8d35055402256078d281a\": container with ID starting with 80b458e6c5ae22ef786fc5dfd40629fce29ea8d189c8d35055402256078d281a not found: ID does not exist"
Jul 30 00:26:42.676890 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:42.676862    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z"]
Jul 30 00:26:42.684948 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:42.684924    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-qxp7z"]
Jul 30 00:26:43.325419 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:43.323675    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e56efbdc-400c-4671-bf3d-e08658f36c08" path="/var/lib/kubelet/pods/e56efbdc-400c-4671-bf3d-e08658f36c08/volumes"
Jul 30 00:26:48.243335 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:48.243252    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhjmv"]
Jul 30 00:26:48.243880 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:48.243609    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e56efbdc-400c-4671-bf3d-e08658f36c08" containerName="manager"
Jul 30 00:26:48.243880 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:48.243623    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e56efbdc-400c-4671-bf3d-e08658f36c08" containerName="manager"
Jul 30 00:26:48.243880 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:48.243680    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e56efbdc-400c-4671-bf3d-e08658f36c08" containerName="manager"
Jul 30 00:26:48.245678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:48.245662    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhjmv"
Jul 30 00:26:48.248328 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:48.248304    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 30 00:26:48.254159 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:48.254135    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhjmv"]
Jul 30 00:26:48.338823 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:48.338791    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhjmv"]
Jul 30 00:26:48.378926 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:48.378892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7886d66f-5822-4115-8a31-ae0e8b394475-config-file\") pod \"limitador-limitador-64c94fd64f-mhjmv\" (UID: \"7886d66f-5822-4115-8a31-ae0e8b394475\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mhjmv"
Jul 30 00:26:48.378926 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:48.378926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68cmw\" (UniqueName: \"kubernetes.io/projected/7886d66f-5822-4115-8a31-ae0e8b394475-kube-api-access-68cmw\") pod \"limitador-limitador-64c94fd64f-mhjmv\" (UID: \"7886d66f-5822-4115-8a31-ae0e8b394475\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mhjmv"
Jul 30 00:26:48.480560 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:48.480521    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7886d66f-5822-4115-8a31-ae0e8b394475-config-file\") pod \"limitador-limitador-64c94fd64f-mhjmv\" (UID: \"7886d66f-5822-4115-8a31-ae0e8b394475\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mhjmv"
Jul 30 00:26:48.480764 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:48.480672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68cmw\" (UniqueName: \"kubernetes.io/projected/7886d66f-5822-4115-8a31-ae0e8b394475-kube-api-access-68cmw\") pod \"limitador-limitador-64c94fd64f-mhjmv\" (UID: \"7886d66f-5822-4115-8a31-ae0e8b394475\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mhjmv"
Jul 30 00:26:48.485189 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:48.481777    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7886d66f-5822-4115-8a31-ae0e8b394475-config-file\") pod \"limitador-limitador-64c94fd64f-mhjmv\" (UID: \"7886d66f-5822-4115-8a31-ae0e8b394475\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mhjmv"
Jul 30 00:26:48.489789 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:48.489762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68cmw\" (UniqueName: \"kubernetes.io/projected/7886d66f-5822-4115-8a31-ae0e8b394475-kube-api-access-68cmw\") pod \"limitador-limitador-64c94fd64f-mhjmv\" (UID: \"7886d66f-5822-4115-8a31-ae0e8b394475\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mhjmv"
Jul 30 00:26:48.556652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:48.556549    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhjmv"
Jul 30 00:26:48.689797 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:48.689770    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhjmv"]
Jul 30 00:26:48.692063 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:26:48.692036    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7886d66f_5822_4115_8a31_ae0e8b394475.slice/crio-cb00770d3ba73a3271f4045bdd942d6401253664c9150d46972ba850884d695f WatchSource:0}: Error finding container cb00770d3ba73a3271f4045bdd942d6401253664c9150d46972ba850884d695f: Status 404 returned error can't find the container with id cb00770d3ba73a3271f4045bdd942d6401253664c9150d46972ba850884d695f
Jul 30 00:26:49.028184 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:49.028146    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-wgc2r"]
Jul 30 00:26:49.031278 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:49.031259    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wgc2r"
Jul 30 00:26:49.034010 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:49.033989    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cjb69\""
Jul 30 00:26:49.039808 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:49.039780    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-wgc2r"]
Jul 30 00:26:49.187172 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:49.187143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlssp\" (UniqueName: \"kubernetes.io/projected/027235bc-197a-48fe-be9f-507a6f463c49-kube-api-access-jlssp\") pod \"authorino-58cd955675-wgc2r\" (UID: \"027235bc-197a-48fe-be9f-507a6f463c49\") " pod="kuadrant-system/authorino-58cd955675-wgc2r"
Jul 30 00:26:49.288500 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:49.288418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlssp\" (UniqueName: \"kubernetes.io/projected/027235bc-197a-48fe-be9f-507a6f463c49-kube-api-access-jlssp\") pod \"authorino-58cd955675-wgc2r\" (UID: \"027235bc-197a-48fe-be9f-507a6f463c49\") " pod="kuadrant-system/authorino-58cd955675-wgc2r"
Jul 30 00:26:49.297460 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:49.297403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlssp\" (UniqueName: \"kubernetes.io/projected/027235bc-197a-48fe-be9f-507a6f463c49-kube-api-access-jlssp\") pod \"authorino-58cd955675-wgc2r\" (UID: \"027235bc-197a-48fe-be9f-507a6f463c49\") " pod="kuadrant-system/authorino-58cd955675-wgc2r"
Jul 30 00:26:49.342405 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:49.342105    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wgc2r"
Jul 30 00:26:49.527401 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:49.527356    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-wgc2r"]
Jul 30 00:26:49.532072 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:26:49.532030    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod027235bc_197a_48fe_be9f_507a6f463c49.slice/crio-00a3e259e264a57ff01e50d25c76e080891ba644f2c3ccf2360f86c77a0afe3d WatchSource:0}: Error finding container 00a3e259e264a57ff01e50d25c76e080891ba644f2c3ccf2360f86c77a0afe3d: Status 404 returned error can't find the container with id 00a3e259e264a57ff01e50d25c76e080891ba644f2c3ccf2360f86c77a0afe3d
Jul 30 00:26:49.693276 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:49.693214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhjmv" event={"ID":"7886d66f-5822-4115-8a31-ae0e8b394475","Type":"ContainerStarted","Data":"cb00770d3ba73a3271f4045bdd942d6401253664c9150d46972ba850884d695f"}
Jul 30 00:26:49.694570 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:49.694530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wgc2r" event={"ID":"027235bc-197a-48fe-be9f-507a6f463c49","Type":"ContainerStarted","Data":"00a3e259e264a57ff01e50d25c76e080891ba644f2c3ccf2360f86c77a0afe3d"}
Jul 30 00:26:52.714899 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:52.714852    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhjmv" event={"ID":"7886d66f-5822-4115-8a31-ae0e8b394475","Type":"ContainerStarted","Data":"2a7a0ef1bf19e5ed41863cd7d67c2af7834eab0b55bb2a91aa8eb28ebabe7d84"}
Jul 30 00:26:52.715403 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:52.714919    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhjmv"
Jul 30 00:26:52.734435 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:52.734382    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhjmv" podStartSLOduration=1.642042716 podStartE2EDuration="4.734359719s" totalImagesPullingTime="3.092317003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:26:52.733385018 +0000 UTC m=+538.030651949" watchObservedRunningTime="2026-07-30 00:26:52.734359719 +0000 UTC m=+538.031626652"
Jul 30 00:26:53.166824 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:53.166790    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-wgc2r"]
Jul 30 00:26:53.721426 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:53.721317    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wgc2r" event={"ID":"027235bc-197a-48fe-be9f-507a6f463c49","Type":"ContainerStarted","Data":"71e88b98050defbf1a3f16608a7c1018b8eae3a2227faca69ceb4c58cf852cae"}
Jul 30 00:26:53.721426 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:53.721410    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-wgc2r" podUID="027235bc-197a-48fe-be9f-507a6f463c49" containerName="authorino" containerID="cri-o://71e88b98050defbf1a3f16608a7c1018b8eae3a2227faca69ceb4c58cf852cae" gracePeriod=30
Jul 30 00:26:53.736191 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:53.736146    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-wgc2r" podStartSLOduration=0.816646585 podStartE2EDuration="4.73612976s" totalImagesPullingTime="3.919483175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:26:53.735511506 +0000 UTC m=+539.032778444" watchObservedRunningTime="2026-07-30 00:26:53.73612976 +0000 UTC m=+539.033396692"
Jul 30 00:26:53.979654 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:53.979590    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wgc2r"
Jul 30 00:26:54.144975 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:54.144942    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlssp\" (UniqueName: \"kubernetes.io/projected/027235bc-197a-48fe-be9f-507a6f463c49-kube-api-access-jlssp\") pod \"027235bc-197a-48fe-be9f-507a6f463c49\" (UID: \"027235bc-197a-48fe-be9f-507a6f463c49\") "
Jul 30 00:26:54.147047 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:54.147022    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027235bc-197a-48fe-be9f-507a6f463c49-kube-api-access-jlssp" (OuterVolumeSpecName: "kube-api-access-jlssp") pod "027235bc-197a-48fe-be9f-507a6f463c49" (UID: "027235bc-197a-48fe-be9f-507a6f463c49"). InnerVolumeSpecName "kube-api-access-jlssp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:26:54.246268 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:54.246176    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlssp\" (UniqueName: \"kubernetes.io/projected/027235bc-197a-48fe-be9f-507a6f463c49-kube-api-access-jlssp\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:26:54.733121 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:54.733080    2579 generic.go:356] "Generic (PLEG): container finished" podID="027235bc-197a-48fe-be9f-507a6f463c49" containerID="71e88b98050defbf1a3f16608a7c1018b8eae3a2227faca69ceb4c58cf852cae" exitCode=0
Jul 30 00:26:54.733646 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:54.733133    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wgc2r"
Jul 30 00:26:54.733646 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:54.733157    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wgc2r" event={"ID":"027235bc-197a-48fe-be9f-507a6f463c49","Type":"ContainerDied","Data":"71e88b98050defbf1a3f16608a7c1018b8eae3a2227faca69ceb4c58cf852cae"}
Jul 30 00:26:54.733646 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:54.733199    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wgc2r" event={"ID":"027235bc-197a-48fe-be9f-507a6f463c49","Type":"ContainerDied","Data":"00a3e259e264a57ff01e50d25c76e080891ba644f2c3ccf2360f86c77a0afe3d"}
Jul 30 00:26:54.733646 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:54.733216    2579 scope.go:117] "RemoveContainer" containerID="71e88b98050defbf1a3f16608a7c1018b8eae3a2227faca69ceb4c58cf852cae"
Jul 30 00:26:54.741820 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:54.741801    2579 scope.go:117] "RemoveContainer" containerID="71e88b98050defbf1a3f16608a7c1018b8eae3a2227faca69ceb4c58cf852cae"
Jul 30 00:26:54.742090 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:26:54.742071    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e88b98050defbf1a3f16608a7c1018b8eae3a2227faca69ceb4c58cf852cae\": container with ID starting with 71e88b98050defbf1a3f16608a7c1018b8eae3a2227faca69ceb4c58cf852cae not found: ID does not exist" containerID="71e88b98050defbf1a3f16608a7c1018b8eae3a2227faca69ceb4c58cf852cae"
Jul 30 00:26:54.742136 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:54.742099    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e88b98050defbf1a3f16608a7c1018b8eae3a2227faca69ceb4c58cf852cae"} err="failed to get container status \"71e88b98050defbf1a3f16608a7c1018b8eae3a2227faca69ceb4c58cf852cae\": rpc error: code = NotFound desc = could not find container \"71e88b98050defbf1a3f16608a7c1018b8eae3a2227faca69ceb4c58cf852cae\": container with ID starting with 71e88b98050defbf1a3f16608a7c1018b8eae3a2227faca69ceb4c58cf852cae not found: ID does not exist"
Jul 30 00:26:54.753864 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:54.753839    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-wgc2r"]
Jul 30 00:26:54.757786 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:54.757767    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-wgc2r"]
Jul 30 00:26:55.323043 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:26:55.323015    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="027235bc-197a-48fe-be9f-507a6f463c49" path="/var/lib/kubelet/pods/027235bc-197a-48fe-be9f-507a6f463c49/volumes"
Jul 30 00:27:02.567475 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:02.567440    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhjmv"]
Jul 30 00:27:02.567888 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:02.567634    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhjmv" podUID="7886d66f-5822-4115-8a31-ae0e8b394475" containerName="limitador" containerID="cri-o://2a7a0ef1bf19e5ed41863cd7d67c2af7834eab0b55bb2a91aa8eb28ebabe7d84" gracePeriod=30
Jul 30 00:27:02.568316 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:02.568282    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhjmv"
Jul 30 00:27:03.115662 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:03.115636    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhjmv"
Jul 30 00:27:03.223771 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:03.223689    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68cmw\" (UniqueName: \"kubernetes.io/projected/7886d66f-5822-4115-8a31-ae0e8b394475-kube-api-access-68cmw\") pod \"7886d66f-5822-4115-8a31-ae0e8b394475\" (UID: \"7886d66f-5822-4115-8a31-ae0e8b394475\") "
Jul 30 00:27:03.223771 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:03.223728    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7886d66f-5822-4115-8a31-ae0e8b394475-config-file\") pod \"7886d66f-5822-4115-8a31-ae0e8b394475\" (UID: \"7886d66f-5822-4115-8a31-ae0e8b394475\") "
Jul 30 00:27:03.224106 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:03.224082    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7886d66f-5822-4115-8a31-ae0e8b394475-config-file" (OuterVolumeSpecName: "config-file") pod "7886d66f-5822-4115-8a31-ae0e8b394475" (UID: "7886d66f-5822-4115-8a31-ae0e8b394475"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:27:03.225841 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:03.225816    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7886d66f-5822-4115-8a31-ae0e8b394475-kube-api-access-68cmw" (OuterVolumeSpecName: "kube-api-access-68cmw") pod "7886d66f-5822-4115-8a31-ae0e8b394475" (UID: "7886d66f-5822-4115-8a31-ae0e8b394475"). InnerVolumeSpecName "kube-api-access-68cmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:27:03.324353 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:03.324326    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68cmw\" (UniqueName: \"kubernetes.io/projected/7886d66f-5822-4115-8a31-ae0e8b394475-kube-api-access-68cmw\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:27:03.324353 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:03.324353    2579 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7886d66f-5822-4115-8a31-ae0e8b394475-config-file\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:27:03.775093 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:03.775058    2579 generic.go:356] "Generic (PLEG): container finished" podID="7886d66f-5822-4115-8a31-ae0e8b394475" containerID="2a7a0ef1bf19e5ed41863cd7d67c2af7834eab0b55bb2a91aa8eb28ebabe7d84" exitCode=0
Jul 30 00:27:03.775609 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:03.775108    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhjmv"
Jul 30 00:27:03.775609 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:03.775143    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhjmv" event={"ID":"7886d66f-5822-4115-8a31-ae0e8b394475","Type":"ContainerDied","Data":"2a7a0ef1bf19e5ed41863cd7d67c2af7834eab0b55bb2a91aa8eb28ebabe7d84"}
Jul 30 00:27:03.775609 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:03.775195    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhjmv" event={"ID":"7886d66f-5822-4115-8a31-ae0e8b394475","Type":"ContainerDied","Data":"cb00770d3ba73a3271f4045bdd942d6401253664c9150d46972ba850884d695f"}
Jul 30 00:27:03.775609 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:03.775220    2579 scope.go:117] "RemoveContainer" containerID="2a7a0ef1bf19e5ed41863cd7d67c2af7834eab0b55bb2a91aa8eb28ebabe7d84"
Jul 30 00:27:03.783620 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:03.783605    2579 scope.go:117] "RemoveContainer" containerID="2a7a0ef1bf19e5ed41863cd7d67c2af7834eab0b55bb2a91aa8eb28ebabe7d84"
Jul 30 00:27:03.783870 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:27:03.783844    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a7a0ef1bf19e5ed41863cd7d67c2af7834eab0b55bb2a91aa8eb28ebabe7d84\": container with ID starting with 2a7a0ef1bf19e5ed41863cd7d67c2af7834eab0b55bb2a91aa8eb28ebabe7d84 not found: ID does not exist" containerID="2a7a0ef1bf19e5ed41863cd7d67c2af7834eab0b55bb2a91aa8eb28ebabe7d84"
Jul 30 00:27:03.783917 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:03.783870    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a7a0ef1bf19e5ed41863cd7d67c2af7834eab0b55bb2a91aa8eb28ebabe7d84"} err="failed to get container status \"2a7a0ef1bf19e5ed41863cd7d67c2af7834eab0b55bb2a91aa8eb28ebabe7d84\": rpc error: code = NotFound desc = could not find container \"2a7a0ef1bf19e5ed41863cd7d67c2af7834eab0b55bb2a91aa8eb28ebabe7d84\": container with ID starting with 2a7a0ef1bf19e5ed41863cd7d67c2af7834eab0b55bb2a91aa8eb28ebabe7d84 not found: ID does not exist"
Jul 30 00:27:03.791749 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:03.791724    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhjmv"]
Jul 30 00:27:03.794041 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:03.794020    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhjmv"]
Jul 30 00:27:04.496283 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.496237    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zpd9x"]
Jul 30 00:27:04.496626 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.496613    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7886d66f-5822-4115-8a31-ae0e8b394475" containerName="limitador"
Jul 30 00:27:04.496669 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.496629    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7886d66f-5822-4115-8a31-ae0e8b394475" containerName="limitador"
Jul 30 00:27:04.496669 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.496651    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="027235bc-197a-48fe-be9f-507a6f463c49" containerName="authorino"
Jul 30 00:27:04.496669 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.496658    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="027235bc-197a-48fe-be9f-507a6f463c49" containerName="authorino"
Jul 30 00:27:04.496763 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.496717    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="027235bc-197a-48fe-be9f-507a6f463c49" containerName="authorino"
Jul 30 00:27:04.496763 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.496728    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7886d66f-5822-4115-8a31-ae0e8b394475" containerName="limitador"
Jul 30 00:27:04.499498 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.499481    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zpd9x"
Jul 30 00:27:04.502308 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.502280    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 30 00:27:04.502448 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.502314    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 30 00:27:04.503554 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.503536    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-wdn4k\""
Jul 30 00:27:04.503632 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.503562    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 30 00:27:04.506687 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.506667    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zpd9x"]
Jul 30 00:27:04.534738 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.534716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/83d6e9fd-c623-479a-94fc-07134cdd40b8-data\") pod \"postgres-69fb4594cf-zpd9x\" (UID: \"83d6e9fd-c623-479a-94fc-07134cdd40b8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zpd9x"
Jul 30 00:27:04.534840 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.534766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scgtg\" (UniqueName: \"kubernetes.io/projected/83d6e9fd-c623-479a-94fc-07134cdd40b8-kube-api-access-scgtg\") pod \"postgres-69fb4594cf-zpd9x\" (UID: \"83d6e9fd-c623-479a-94fc-07134cdd40b8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zpd9x"
Jul 30 00:27:04.635967 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.635941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/83d6e9fd-c623-479a-94fc-07134cdd40b8-data\") pod \"postgres-69fb4594cf-zpd9x\" (UID: \"83d6e9fd-c623-479a-94fc-07134cdd40b8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zpd9x"
Jul 30 00:27:04.636111 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.636029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scgtg\" (UniqueName: \"kubernetes.io/projected/83d6e9fd-c623-479a-94fc-07134cdd40b8-kube-api-access-scgtg\") pod \"postgres-69fb4594cf-zpd9x\" (UID: \"83d6e9fd-c623-479a-94fc-07134cdd40b8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zpd9x"
Jul 30 00:27:04.636380 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.636361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/83d6e9fd-c623-479a-94fc-07134cdd40b8-data\") pod \"postgres-69fb4594cf-zpd9x\" (UID: \"83d6e9fd-c623-479a-94fc-07134cdd40b8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zpd9x"
Jul 30 00:27:04.644515 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.644497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scgtg\" (UniqueName: \"kubernetes.io/projected/83d6e9fd-c623-479a-94fc-07134cdd40b8-kube-api-access-scgtg\") pod \"postgres-69fb4594cf-zpd9x\" (UID: \"83d6e9fd-c623-479a-94fc-07134cdd40b8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zpd9x"
Jul 30 00:27:04.811305 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.811214    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zpd9x"
Jul 30 00:27:04.946266 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:04.946222    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zpd9x"]
Jul 30 00:27:04.948399 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:27:04.948371    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d6e9fd_c623_479a_94fc_07134cdd40b8.slice/crio-d4f741959a8d8cffe699bab217f0c6b45f09ea4c8d1a8ee9f20db2a97c48d4ec WatchSource:0}: Error finding container d4f741959a8d8cffe699bab217f0c6b45f09ea4c8d1a8ee9f20db2a97c48d4ec: Status 404 returned error can't find the container with id d4f741959a8d8cffe699bab217f0c6b45f09ea4c8d1a8ee9f20db2a97c48d4ec
Jul 30 00:27:05.322305 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:05.322273    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7886d66f-5822-4115-8a31-ae0e8b394475" path="/var/lib/kubelet/pods/7886d66f-5822-4115-8a31-ae0e8b394475/volumes"
Jul 30 00:27:05.786993 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:05.786954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zpd9x" event={"ID":"83d6e9fd-c623-479a-94fc-07134cdd40b8","Type":"ContainerStarted","Data":"d4f741959a8d8cffe699bab217f0c6b45f09ea4c8d1a8ee9f20db2a97c48d4ec"}
Jul 30 00:27:10.813614 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:10.813579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zpd9x" event={"ID":"83d6e9fd-c623-479a-94fc-07134cdd40b8","Type":"ContainerStarted","Data":"61f27be753c1ee932616d9919d10791fe1e6baad59a2958c35e6ae25b1e5c8d1"}
Jul 30 00:27:10.814010 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:10.813666    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zpd9x"
Jul 30 00:27:10.829574 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:10.829527    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zpd9x" podStartSLOduration=1.887596842 podStartE2EDuration="6.829514157s" totalImagesPullingTime="4.941917315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:27:10.828285052 +0000 UTC m=+556.125551984" watchObservedRunningTime="2026-07-30 00:27:10.829514157 +0000 UTC m=+556.126781088"
Jul 30 00:27:16.847476 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:16.847445    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zpd9x"
Jul 30 00:27:17.354025 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.353990    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tx7tx"]
Jul 30 00:27:17.359463 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.359438    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tx7tx"
Jul 30 00:27:17.362385 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.362352    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cjb69\""
Jul 30 00:27:17.364729 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.364692    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tx7tx"]
Jul 30 00:27:17.455172 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.455141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t8x7\" (UniqueName: \"kubernetes.io/projected/1280ae17-e098-420f-82a9-0a1bdd809bee-kube-api-access-9t8x7\") pod \"authorino-5d4b67c446-tx7tx\" (UID: \"1280ae17-e098-420f-82a9-0a1bdd809bee\") " pod="kuadrant-system/authorino-5d4b67c446-tx7tx"
Jul 30 00:27:17.556659 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.556629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t8x7\" (UniqueName: \"kubernetes.io/projected/1280ae17-e098-420f-82a9-0a1bdd809bee-kube-api-access-9t8x7\") pod \"authorino-5d4b67c446-tx7tx\" (UID: \"1280ae17-e098-420f-82a9-0a1bdd809bee\") " pod="kuadrant-system/authorino-5d4b67c446-tx7tx"
Jul 30 00:27:17.564854 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.564815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t8x7\" (UniqueName: \"kubernetes.io/projected/1280ae17-e098-420f-82a9-0a1bdd809bee-kube-api-access-9t8x7\") pod \"authorino-5d4b67c446-tx7tx\" (UID: \"1280ae17-e098-420f-82a9-0a1bdd809bee\") " pod="kuadrant-system/authorino-5d4b67c446-tx7tx"
Jul 30 00:27:17.575806 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.575780    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tx7tx"]
Jul 30 00:27:17.576039 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.576027    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tx7tx"
Jul 30 00:27:17.600235 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.600212    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-597b759896-b5h2m"]
Jul 30 00:27:17.605266 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.605187    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-597b759896-b5h2m"
Jul 30 00:27:17.610626 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.610601    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-597b759896-b5h2m"]
Jul 30 00:27:17.657704 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.657677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcz4k\" (UniqueName: \"kubernetes.io/projected/41ae0b60-e908-4fcd-9848-81c1a9ab0cbe-kube-api-access-hcz4k\") pod \"authorino-597b759896-b5h2m\" (UID: \"41ae0b60-e908-4fcd-9848-81c1a9ab0cbe\") " pod="kuadrant-system/authorino-597b759896-b5h2m"
Jul 30 00:27:17.708356 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.708321    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-597b759896-b5h2m"]
Jul 30 00:27:17.708736 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:27:17.708710    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hcz4k], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-597b759896-b5h2m" podUID="41ae0b60-e908-4fcd-9848-81c1a9ab0cbe"
Jul 30 00:27:17.721781 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.721762    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tx7tx"]
Jul 30 00:27:17.724162 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:27:17.724136    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1280ae17_e098_420f_82a9_0a1bdd809bee.slice/crio-0da4004a544ef7f97c3d95f1eea1b4c20ad11d4bdff2e29062b350041ac17472 WatchSource:0}: Error finding container 0da4004a544ef7f97c3d95f1eea1b4c20ad11d4bdff2e29062b350041ac17472: Status 404 returned error can't find the container with id 0da4004a544ef7f97c3d95f1eea1b4c20ad11d4bdff2e29062b350041ac17472
Jul 30 00:27:17.734465 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.734443    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7fddc5c74d-qcfng"]
Jul 30 00:27:17.745604 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.745579    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fddc5c74d-qcfng"
Jul 30 00:27:17.746542 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.746514    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7fddc5c74d-qcfng"]
Jul 30 00:27:17.748735 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.748711    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 30 00:27:17.758445 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.758422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcz4k\" (UniqueName: \"kubernetes.io/projected/41ae0b60-e908-4fcd-9848-81c1a9ab0cbe-kube-api-access-hcz4k\") pod \"authorino-597b759896-b5h2m\" (UID: \"41ae0b60-e908-4fcd-9848-81c1a9ab0cbe\") " pod="kuadrant-system/authorino-597b759896-b5h2m"
Jul 30 00:27:17.768066 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.768044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcz4k\" (UniqueName: \"kubernetes.io/projected/41ae0b60-e908-4fcd-9848-81c1a9ab0cbe-kube-api-access-hcz4k\") pod \"authorino-597b759896-b5h2m\" (UID: \"41ae0b60-e908-4fcd-9848-81c1a9ab0cbe\") " pod="kuadrant-system/authorino-597b759896-b5h2m"
Jul 30 00:27:17.844909 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.844874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tx7tx" event={"ID":"1280ae17-e098-420f-82a9-0a1bdd809bee","Type":"ContainerStarted","Data":"0da4004a544ef7f97c3d95f1eea1b4c20ad11d4bdff2e29062b350041ac17472"}
Jul 30 00:27:17.844909 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.844908    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-597b759896-b5h2m"
Jul 30 00:27:17.852285 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.852264    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-597b759896-b5h2m"
Jul 30 00:27:17.858978 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.858925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmgqx\" (UniqueName: \"kubernetes.io/projected/1837fa42-9342-468b-bc78-58900786eaf1-kube-api-access-tmgqx\") pod \"authorino-7fddc5c74d-qcfng\" (UID: \"1837fa42-9342-468b-bc78-58900786eaf1\") " pod="kuadrant-system/authorino-7fddc5c74d-qcfng"
Jul 30 00:27:17.859081 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.859037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1837fa42-9342-468b-bc78-58900786eaf1-tls-cert\") pod \"authorino-7fddc5c74d-qcfng\" (UID: \"1837fa42-9342-468b-bc78-58900786eaf1\") " pod="kuadrant-system/authorino-7fddc5c74d-qcfng"
Jul 30 00:27:17.960390 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.960358    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcz4k\" (UniqueName: \"kubernetes.io/projected/41ae0b60-e908-4fcd-9848-81c1a9ab0cbe-kube-api-access-hcz4k\") pod \"41ae0b60-e908-4fcd-9848-81c1a9ab0cbe\" (UID: \"41ae0b60-e908-4fcd-9848-81c1a9ab0cbe\") "
Jul 30 00:27:17.960667 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.960644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmgqx\" (UniqueName: \"kubernetes.io/projected/1837fa42-9342-468b-bc78-58900786eaf1-kube-api-access-tmgqx\") pod \"authorino-7fddc5c74d-qcfng\" (UID: \"1837fa42-9342-468b-bc78-58900786eaf1\") " pod="kuadrant-system/authorino-7fddc5c74d-qcfng"
Jul 30 00:27:17.960742 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.960725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1837fa42-9342-468b-bc78-58900786eaf1-tls-cert\") pod \"authorino-7fddc5c74d-qcfng\" (UID: \"1837fa42-9342-468b-bc78-58900786eaf1\") " pod="kuadrant-system/authorino-7fddc5c74d-qcfng"
Jul 30 00:27:17.962629 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.962602    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ae0b60-e908-4fcd-9848-81c1a9ab0cbe-kube-api-access-hcz4k" (OuterVolumeSpecName: "kube-api-access-hcz4k") pod "41ae0b60-e908-4fcd-9848-81c1a9ab0cbe" (UID: "41ae0b60-e908-4fcd-9848-81c1a9ab0cbe"). InnerVolumeSpecName "kube-api-access-hcz4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:27:17.963043 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.963027    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1837fa42-9342-468b-bc78-58900786eaf1-tls-cert\") pod \"authorino-7fddc5c74d-qcfng\" (UID: \"1837fa42-9342-468b-bc78-58900786eaf1\") " pod="kuadrant-system/authorino-7fddc5c74d-qcfng"
Jul 30 00:27:17.968540 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:17.968517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmgqx\" (UniqueName: \"kubernetes.io/projected/1837fa42-9342-468b-bc78-58900786eaf1-kube-api-access-tmgqx\") pod \"authorino-7fddc5c74d-qcfng\" (UID: \"1837fa42-9342-468b-bc78-58900786eaf1\") " pod="kuadrant-system/authorino-7fddc5c74d-qcfng"
Jul 30 00:27:18.056646 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:18.056618    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fddc5c74d-qcfng"
Jul 30 00:27:18.062038 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:18.062017    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcz4k\" (UniqueName: \"kubernetes.io/projected/41ae0b60-e908-4fcd-9848-81c1a9ab0cbe-kube-api-access-hcz4k\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:27:18.198797 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:18.198767    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7fddc5c74d-qcfng"]
Jul 30 00:27:18.200757 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:27:18.200733    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1837fa42_9342_468b_bc78_58900786eaf1.slice/crio-e88e92fad0e4dd5eda59db144aef782baf432031e5d6ee2919f78292f0d91197 WatchSource:0}: Error finding container e88e92fad0e4dd5eda59db144aef782baf432031e5d6ee2919f78292f0d91197: Status 404 returned error can't find the container with id e88e92fad0e4dd5eda59db144aef782baf432031e5d6ee2919f78292f0d91197
Jul 30 00:27:18.852372 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:18.852272    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tx7tx" event={"ID":"1280ae17-e098-420f-82a9-0a1bdd809bee","Type":"ContainerStarted","Data":"1ae63195ec6b07eba9c8c42c6597e96157eb7b7ec474cdf8ef769a7a154778c7"}
Jul 30 00:27:18.852372 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:18.852349    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-tx7tx" podUID="1280ae17-e098-420f-82a9-0a1bdd809bee" containerName="authorino" containerID="cri-o://1ae63195ec6b07eba9c8c42c6597e96157eb7b7ec474cdf8ef769a7a154778c7" gracePeriod=30
Jul 30 00:27:18.853909 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:18.853883    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-597b759896-b5h2m"
Jul 30 00:27:18.854052 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:18.853877    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fddc5c74d-qcfng" event={"ID":"1837fa42-9342-468b-bc78-58900786eaf1","Type":"ContainerStarted","Data":"87761dcc38b8ad033e39e0fa36475872f2e44d6ffb54cf92c5bb862e2062d486"}
Jul 30 00:27:18.854052 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:18.854005    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fddc5c74d-qcfng" event={"ID":"1837fa42-9342-468b-bc78-58900786eaf1","Type":"ContainerStarted","Data":"e88e92fad0e4dd5eda59db144aef782baf432031e5d6ee2919f78292f0d91197"}
Jul 30 00:27:18.866782 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:18.866743    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-tx7tx" podStartSLOduration=1.352746345 podStartE2EDuration="1.866730872s" totalImagesPullingTime="513.984527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:27:18.866291456 +0000 UTC m=+564.163558388" watchObservedRunningTime="2026-07-30 00:27:18.866730872 +0000 UTC m=+564.163997803"
Jul 30 00:27:18.893619 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:18.893595    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-597b759896-b5h2m"]
Jul 30 00:27:18.897105 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:18.897085    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-597b759896-b5h2m"]
Jul 30 00:27:18.910860 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:18.910814    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7fddc5c74d-qcfng" podStartSLOduration=1.483370558 podStartE2EDuration="1.910800221s" totalImagesPullingTime="427.429663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:27:18.909380725 +0000 UTC m=+564.206647657" watchObservedRunningTime="2026-07-30 00:27:18.910800221 +0000 UTC m=+564.208067152"
Jul 30 00:27:19.131653 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:19.131626    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tx7tx"
Jul 30 00:27:19.271818 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:19.271789    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t8x7\" (UniqueName: \"kubernetes.io/projected/1280ae17-e098-420f-82a9-0a1bdd809bee-kube-api-access-9t8x7\") pod \"1280ae17-e098-420f-82a9-0a1bdd809bee\" (UID: \"1280ae17-e098-420f-82a9-0a1bdd809bee\") "
Jul 30 00:27:19.273859 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:19.273831    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1280ae17-e098-420f-82a9-0a1bdd809bee-kube-api-access-9t8x7" (OuterVolumeSpecName: "kube-api-access-9t8x7") pod "1280ae17-e098-420f-82a9-0a1bdd809bee" (UID: "1280ae17-e098-420f-82a9-0a1bdd809bee"). InnerVolumeSpecName "kube-api-access-9t8x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:27:19.322383 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:19.322352    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41ae0b60-e908-4fcd-9848-81c1a9ab0cbe" path="/var/lib/kubelet/pods/41ae0b60-e908-4fcd-9848-81c1a9ab0cbe/volumes"
Jul 30 00:27:19.373129 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:19.373104    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t8x7\" (UniqueName: \"kubernetes.io/projected/1280ae17-e098-420f-82a9-0a1bdd809bee-kube-api-access-9t8x7\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:27:19.860096 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:19.860016    2579 generic.go:356] "Generic (PLEG): container finished" podID="1280ae17-e098-420f-82a9-0a1bdd809bee" containerID="1ae63195ec6b07eba9c8c42c6597e96157eb7b7ec474cdf8ef769a7a154778c7" exitCode=0
Jul 30 00:27:19.860096 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:19.860061    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tx7tx"
Jul 30 00:27:19.860664 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:19.860095    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tx7tx" event={"ID":"1280ae17-e098-420f-82a9-0a1bdd809bee","Type":"ContainerDied","Data":"1ae63195ec6b07eba9c8c42c6597e96157eb7b7ec474cdf8ef769a7a154778c7"}
Jul 30 00:27:19.860664 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:19.860132    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tx7tx" event={"ID":"1280ae17-e098-420f-82a9-0a1bdd809bee","Type":"ContainerDied","Data":"0da4004a544ef7f97c3d95f1eea1b4c20ad11d4bdff2e29062b350041ac17472"}
Jul 30 00:27:19.860664 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:19.860150    2579 scope.go:117] "RemoveContainer" containerID="1ae63195ec6b07eba9c8c42c6597e96157eb7b7ec474cdf8ef769a7a154778c7"
Jul 30 00:27:19.868836 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:19.868817    2579 scope.go:117] "RemoveContainer" containerID="1ae63195ec6b07eba9c8c42c6597e96157eb7b7ec474cdf8ef769a7a154778c7"
Jul 30 00:27:19.869100 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:27:19.869079    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ae63195ec6b07eba9c8c42c6597e96157eb7b7ec474cdf8ef769a7a154778c7\": container with ID starting with 1ae63195ec6b07eba9c8c42c6597e96157eb7b7ec474cdf8ef769a7a154778c7 not found: ID does not exist" containerID="1ae63195ec6b07eba9c8c42c6597e96157eb7b7ec474cdf8ef769a7a154778c7"
Jul 30 00:27:19.869177 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:19.869113    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ae63195ec6b07eba9c8c42c6597e96157eb7b7ec474cdf8ef769a7a154778c7"} err="failed to get container status \"1ae63195ec6b07eba9c8c42c6597e96157eb7b7ec474cdf8ef769a7a154778c7\": rpc error: code = NotFound desc = could not find container \"1ae63195ec6b07eba9c8c42c6597e96157eb7b7ec474cdf8ef769a7a154778c7\": container with ID starting with 1ae63195ec6b07eba9c8c42c6597e96157eb7b7ec474cdf8ef769a7a154778c7 not found: ID does not exist"
Jul 30 00:27:19.877796 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:19.877773    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tx7tx"]
Jul 30 00:27:19.881641 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:19.881618    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tx7tx"]
Jul 30 00:27:21.322511 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:21.322480    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1280ae17-e098-420f-82a9-0a1bdd809bee" path="/var/lib/kubelet/pods/1280ae17-e098-420f-82a9-0a1bdd809bee/volumes"
Jul 30 00:27:23.324372 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:23.324337    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b4476f574-rpnhh"]
Jul 30 00:27:23.324871 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:23.324851    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1280ae17-e098-420f-82a9-0a1bdd809bee" containerName="authorino"
Jul 30 00:27:23.324944 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:23.324875    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1280ae17-e098-420f-82a9-0a1bdd809bee" containerName="authorino"
Jul 30 00:27:23.324993 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:23.324974    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1280ae17-e098-420f-82a9-0a1bdd809bee" containerName="authorino"
Jul 30 00:27:23.342725 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:23.342698    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b4476f574-rpnhh"]
Jul 30 00:27:23.342887 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:23.342811    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b4476f574-rpnhh"
Jul 30 00:27:23.345614 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:23.345586    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 30 00:27:23.345759 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:23.345643    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-94z2g\""
Jul 30 00:27:23.345759 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:23.345646    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 30 00:27:23.406071 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:23.406045    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7188f35c-e705-4b6c-ad6f-9afda11899da-webhook-cert\") pod \"maas-controller-6b4476f574-rpnhh\" (UID: \"7188f35c-e705-4b6c-ad6f-9afda11899da\") " pod="opendatahub/maas-controller-6b4476f574-rpnhh"
Jul 30 00:27:23.406214 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:23.406076    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksvvx\" (UniqueName: \"kubernetes.io/projected/7188f35c-e705-4b6c-ad6f-9afda11899da-kube-api-access-ksvvx\") pod \"maas-controller-6b4476f574-rpnhh\" (UID: \"7188f35c-e705-4b6c-ad6f-9afda11899da\") " pod="opendatahub/maas-controller-6b4476f574-rpnhh"
Jul 30 00:27:23.507329 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:23.507303    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7188f35c-e705-4b6c-ad6f-9afda11899da-webhook-cert\") pod \"maas-controller-6b4476f574-rpnhh\" (UID: \"7188f35c-e705-4b6c-ad6f-9afda11899da\") " pod="opendatahub/maas-controller-6b4476f574-rpnhh"
Jul 30 00:27:23.507467 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:23.507334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksvvx\" (UniqueName: \"kubernetes.io/projected/7188f35c-e705-4b6c-ad6f-9afda11899da-kube-api-access-ksvvx\") pod \"maas-controller-6b4476f574-rpnhh\" (UID: \"7188f35c-e705-4b6c-ad6f-9afda11899da\") " pod="opendatahub/maas-controller-6b4476f574-rpnhh"
Jul 30 00:27:23.509866 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:23.509841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7188f35c-e705-4b6c-ad6f-9afda11899da-webhook-cert\") pod \"maas-controller-6b4476f574-rpnhh\" (UID: \"7188f35c-e705-4b6c-ad6f-9afda11899da\") " pod="opendatahub/maas-controller-6b4476f574-rpnhh"
Jul 30 00:27:23.520032 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:23.520002    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksvvx\" (UniqueName: \"kubernetes.io/projected/7188f35c-e705-4b6c-ad6f-9afda11899da-kube-api-access-ksvvx\") pod \"maas-controller-6b4476f574-rpnhh\" (UID: \"7188f35c-e705-4b6c-ad6f-9afda11899da\") " pod="opendatahub/maas-controller-6b4476f574-rpnhh"
Jul 30 00:27:23.654128 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:23.654096    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b4476f574-rpnhh"
Jul 30 00:27:23.814325 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:23.814302    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b4476f574-rpnhh"]
Jul 30 00:27:23.816681 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:27:23.816654    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7188f35c_e705_4b6c_ad6f_9afda11899da.slice/crio-79a3efc47aad834ae38a6b7aaac9a8bc9035bafe49bea4b98a68bc547235e593 WatchSource:0}: Error finding container 79a3efc47aad834ae38a6b7aaac9a8bc9035bafe49bea4b98a68bc547235e593: Status 404 returned error can't find the container with id 79a3efc47aad834ae38a6b7aaac9a8bc9035bafe49bea4b98a68bc547235e593
Jul 30 00:27:23.884294 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:23.884261    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b4476f574-rpnhh" event={"ID":"7188f35c-e705-4b6c-ad6f-9afda11899da","Type":"ContainerStarted","Data":"79a3efc47aad834ae38a6b7aaac9a8bc9035bafe49bea4b98a68bc547235e593"}
Jul 30 00:27:26.902164 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:26.902129    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b4476f574-rpnhh" event={"ID":"7188f35c-e705-4b6c-ad6f-9afda11899da","Type":"ContainerStarted","Data":"5ca00ee79b015d3bae01e877b764a6c309dc6797d2f5df2549e667463a824b3a"}
Jul 30 00:27:26.902608 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:26.902221    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b4476f574-rpnhh"
Jul 30 00:27:26.920612 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:26.920565    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6b4476f574-rpnhh" podStartSLOduration=1.7733917300000002 podStartE2EDuration="3.920548311s" totalImagesPullingTime="2.147156581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:27:26.918743349 +0000 UTC m=+572.216010283" watchObservedRunningTime="2026-07-30 00:27:26.920548311 +0000 UTC m=+572.217815242"
Jul 30 00:27:37.342576 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:37.342544    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b4476f574-rpnhh"]
Jul 30 00:27:37.343060 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:37.342712    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6b4476f574-rpnhh" podUID="7188f35c-e705-4b6c-ad6f-9afda11899da" containerName="manager" containerID="cri-o://5ca00ee79b015d3bae01e877b764a6c309dc6797d2f5df2549e667463a824b3a" gracePeriod=30
Jul 30 00:27:37.344525 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:37.344498    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b4476f574-rpnhh"
Jul 30 00:27:37.589866 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:37.589839    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b4476f574-rpnhh"
Jul 30 00:27:37.735957 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:37.735931    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7188f35c-e705-4b6c-ad6f-9afda11899da-webhook-cert\") pod \"7188f35c-e705-4b6c-ad6f-9afda11899da\" (UID: \"7188f35c-e705-4b6c-ad6f-9afda11899da\") "
Jul 30 00:27:37.736174 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:37.735965    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksvvx\" (UniqueName: \"kubernetes.io/projected/7188f35c-e705-4b6c-ad6f-9afda11899da-kube-api-access-ksvvx\") pod \"7188f35c-e705-4b6c-ad6f-9afda11899da\" (UID: \"7188f35c-e705-4b6c-ad6f-9afda11899da\") "
Jul 30 00:27:37.738139 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:37.738101    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7188f35c-e705-4b6c-ad6f-9afda11899da-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7188f35c-e705-4b6c-ad6f-9afda11899da" (UID: "7188f35c-e705-4b6c-ad6f-9afda11899da"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:27:37.738271 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:37.738191    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7188f35c-e705-4b6c-ad6f-9afda11899da-kube-api-access-ksvvx" (OuterVolumeSpecName: "kube-api-access-ksvvx") pod "7188f35c-e705-4b6c-ad6f-9afda11899da" (UID: "7188f35c-e705-4b6c-ad6f-9afda11899da"). InnerVolumeSpecName "kube-api-access-ksvvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:27:37.837136 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:37.837111    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7188f35c-e705-4b6c-ad6f-9afda11899da-webhook-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:27:37.837136 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:37.837133    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksvvx\" (UniqueName: \"kubernetes.io/projected/7188f35c-e705-4b6c-ad6f-9afda11899da-kube-api-access-ksvvx\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:27:37.950143 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:37.950113    2579 generic.go:356] "Generic (PLEG): container finished" podID="7188f35c-e705-4b6c-ad6f-9afda11899da" containerID="5ca00ee79b015d3bae01e877b764a6c309dc6797d2f5df2549e667463a824b3a" exitCode=0
Jul 30 00:27:37.950302 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:37.950163    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b4476f574-rpnhh"
Jul 30 00:27:37.950302 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:37.950205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b4476f574-rpnhh" event={"ID":"7188f35c-e705-4b6c-ad6f-9afda11899da","Type":"ContainerDied","Data":"5ca00ee79b015d3bae01e877b764a6c309dc6797d2f5df2549e667463a824b3a"}
Jul 30 00:27:37.950302 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:37.950284    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b4476f574-rpnhh" event={"ID":"7188f35c-e705-4b6c-ad6f-9afda11899da","Type":"ContainerDied","Data":"79a3efc47aad834ae38a6b7aaac9a8bc9035bafe49bea4b98a68bc547235e593"}
Jul 30 00:27:37.950479 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:37.950318    2579 scope.go:117] "RemoveContainer" containerID="5ca00ee79b015d3bae01e877b764a6c309dc6797d2f5df2549e667463a824b3a"
Jul 30 00:27:37.959326 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:37.959308    2579 scope.go:117] "RemoveContainer" containerID="5ca00ee79b015d3bae01e877b764a6c309dc6797d2f5df2549e667463a824b3a"
Jul 30 00:27:37.959586 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:27:37.959569    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ca00ee79b015d3bae01e877b764a6c309dc6797d2f5df2549e667463a824b3a\": container with ID starting with 5ca00ee79b015d3bae01e877b764a6c309dc6797d2f5df2549e667463a824b3a not found: ID does not exist" containerID="5ca00ee79b015d3bae01e877b764a6c309dc6797d2f5df2549e667463a824b3a"
Jul 30 00:27:37.959632 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:37.959594    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca00ee79b015d3bae01e877b764a6c309dc6797d2f5df2549e667463a824b3a"} err="failed to get container status \"5ca00ee79b015d3bae01e877b764a6c309dc6797d2f5df2549e667463a824b3a\": rpc error: code = NotFound desc = could not find container \"5ca00ee79b015d3bae01e877b764a6c309dc6797d2f5df2549e667463a824b3a\": container with ID starting with 5ca00ee79b015d3bae01e877b764a6c309dc6797d2f5df2549e667463a824b3a not found: ID does not exist"
Jul 30 00:27:37.970990 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:37.970969    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b4476f574-rpnhh"]
Jul 30 00:27:37.974663 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:37.974644    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b4476f574-rpnhh"]
Jul 30 00:27:39.321525 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:39.321491    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7188f35c-e705-4b6c-ad6f-9afda11899da" path="/var/lib/kubelet/pods/7188f35c-e705-4b6c-ad6f-9afda11899da/volumes"
Jul 30 00:27:55.149489 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:55.149460    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:27:55.156503 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:55.156479    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:27:58.613614 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:58.613578    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-68d9bb9c54-bh2mr"]
Jul 30 00:27:58.614008 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:58.613994    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7188f35c-e705-4b6c-ad6f-9afda11899da" containerName="manager"
Jul 30 00:27:58.614008 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:58.614009    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7188f35c-e705-4b6c-ad6f-9afda11899da" containerName="manager"
Jul 30 00:27:58.614082 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:58.614074    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7188f35c-e705-4b6c-ad6f-9afda11899da" containerName="manager"
Jul 30 00:27:58.617153 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:58.617127    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68d9bb9c54-bh2mr"
Jul 30 00:27:58.620278 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:58.620257    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 30 00:27:58.620278 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:58.620270    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wx5r8\""
Jul 30 00:27:58.629805 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:58.629782    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68d9bb9c54-bh2mr"]
Jul 30 00:27:58.686227 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:58.686202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b0496d-ddb4-4f0f-8684-f8031487b466-maas-api-tls\") pod \"maas-api-68d9bb9c54-bh2mr\" (UID: \"57b0496d-ddb4-4f0f-8684-f8031487b466\") " pod="odh-ai-gateway-infra/maas-api-68d9bb9c54-bh2mr"
Jul 30 00:27:58.686365 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:58.686235    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx2g9\" (UniqueName: \"kubernetes.io/projected/57b0496d-ddb4-4f0f-8684-f8031487b466-kube-api-access-kx2g9\") pod \"maas-api-68d9bb9c54-bh2mr\" (UID: \"57b0496d-ddb4-4f0f-8684-f8031487b466\") " pod="odh-ai-gateway-infra/maas-api-68d9bb9c54-bh2mr"
Jul 30 00:27:58.686365 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:58.686321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b0496d-ddb4-4f0f-8684-f8031487b466-openshift-service-ca\") pod \"maas-api-68d9bb9c54-bh2mr\" (UID: \"57b0496d-ddb4-4f0f-8684-f8031487b466\") " pod="odh-ai-gateway-infra/maas-api-68d9bb9c54-bh2mr"
Jul 30 00:27:58.787447 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:58.787421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b0496d-ddb4-4f0f-8684-f8031487b466-maas-api-tls\") pod \"maas-api-68d9bb9c54-bh2mr\" (UID: \"57b0496d-ddb4-4f0f-8684-f8031487b466\") " pod="odh-ai-gateway-infra/maas-api-68d9bb9c54-bh2mr"
Jul 30 00:27:58.787562 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:58.787453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx2g9\" (UniqueName: \"kubernetes.io/projected/57b0496d-ddb4-4f0f-8684-f8031487b466-kube-api-access-kx2g9\") pod \"maas-api-68d9bb9c54-bh2mr\" (UID: \"57b0496d-ddb4-4f0f-8684-f8031487b466\") " pod="odh-ai-gateway-infra/maas-api-68d9bb9c54-bh2mr"
Jul 30 00:27:58.787562 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:58.787479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b0496d-ddb4-4f0f-8684-f8031487b466-openshift-service-ca\") pod \"maas-api-68d9bb9c54-bh2mr\" (UID: \"57b0496d-ddb4-4f0f-8684-f8031487b466\") " pod="odh-ai-gateway-infra/maas-api-68d9bb9c54-bh2mr"
Jul 30 00:27:58.788068 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:58.788047    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b0496d-ddb4-4f0f-8684-f8031487b466-openshift-service-ca\") pod \"maas-api-68d9bb9c54-bh2mr\" (UID: \"57b0496d-ddb4-4f0f-8684-f8031487b466\") " pod="odh-ai-gateway-infra/maas-api-68d9bb9c54-bh2mr"
Jul 30 00:27:58.790020 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:58.789991    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b0496d-ddb4-4f0f-8684-f8031487b466-maas-api-tls\") pod \"maas-api-68d9bb9c54-bh2mr\" (UID: \"57b0496d-ddb4-4f0f-8684-f8031487b466\") " pod="odh-ai-gateway-infra/maas-api-68d9bb9c54-bh2mr"
Jul 30 00:27:58.800556 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:58.800533    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx2g9\" (UniqueName: \"kubernetes.io/projected/57b0496d-ddb4-4f0f-8684-f8031487b466-kube-api-access-kx2g9\") pod \"maas-api-68d9bb9c54-bh2mr\" (UID: \"57b0496d-ddb4-4f0f-8684-f8031487b466\") " pod="odh-ai-gateway-infra/maas-api-68d9bb9c54-bh2mr"
Jul 30 00:27:58.928066 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:58.928035    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68d9bb9c54-bh2mr"
Jul 30 00:27:59.065773 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.065713    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68d9bb9c54-bh2mr"]
Jul 30 00:27:59.068011 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:27:59.067983    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b0496d_ddb4_4f0f_8684_f8031487b466.slice/crio-1086412bfc521a84920ba9d4addeacd1319bec1c4bf473c2062255110bb3e32b WatchSource:0}: Error finding container 1086412bfc521a84920ba9d4addeacd1319bec1c4bf473c2062255110bb3e32b: Status 404 returned error can't find the container with id 1086412bfc521a84920ba9d4addeacd1319bec1c4bf473c2062255110bb3e32b
Jul 30 00:27:59.102104 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.102079    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-m7n6j"]
Jul 30 00:27:59.108488 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.108469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-m7n6j"
Jul 30 00:27:59.111256 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.111215    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 30 00:27:59.115848 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.115824    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-m7n6j"]
Jul 30 00:27:59.141581 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.141553    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-9k8fv"]
Jul 30 00:27:59.146511 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.146488    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-9k8fv"
Jul 30 00:27:59.149676 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.149655    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2jr67\""
Jul 30 00:27:59.155561 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.155539    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-9k8fv"]
Jul 30 00:27:59.190969 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.190895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7s92\" (UniqueName: \"kubernetes.io/projected/2e048fda-d493-49d2-955e-a4f1bb2a7f8e-kube-api-access-c7s92\") pod \"payload-processing-5cc6cd444b-9k8fv\" (UID: \"2e048fda-d493-49d2-955e-a4f1bb2a7f8e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9k8fv"
Jul 30 00:27:59.190969 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.190933    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e048fda-d493-49d2-955e-a4f1bb2a7f8e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-9k8fv\" (UID: \"2e048fda-d493-49d2-955e-a4f1bb2a7f8e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9k8fv"
Jul 30 00:27:59.191132 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.190996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qdxn\" (UniqueName: \"kubernetes.io/projected/9fc56b79-4db8-4489-a8c5-95e6349ce60d-kube-api-access-5qdxn\") pod \"payload-pre-processing-687dbb6cf6-m7n6j\" (UID: \"9fc56b79-4db8-4489-a8c5-95e6349ce60d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-m7n6j"
Jul 30 00:27:59.191132 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.191023    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc56b79-4db8-4489-a8c5-95e6349ce60d-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-m7n6j\" (UID: \"9fc56b79-4db8-4489-a8c5-95e6349ce60d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-m7n6j"
Jul 30 00:27:59.292390 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.292361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qdxn\" (UniqueName: \"kubernetes.io/projected/9fc56b79-4db8-4489-a8c5-95e6349ce60d-kube-api-access-5qdxn\") pod \"payload-pre-processing-687dbb6cf6-m7n6j\" (UID: \"9fc56b79-4db8-4489-a8c5-95e6349ce60d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-m7n6j"
Jul 30 00:27:59.292543 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.292470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc56b79-4db8-4489-a8c5-95e6349ce60d-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-m7n6j\" (UID: \"9fc56b79-4db8-4489-a8c5-95e6349ce60d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-m7n6j"
Jul 30 00:27:59.292612 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.292562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7s92\" (UniqueName: \"kubernetes.io/projected/2e048fda-d493-49d2-955e-a4f1bb2a7f8e-kube-api-access-c7s92\") pod \"payload-processing-5cc6cd444b-9k8fv\" (UID: \"2e048fda-d493-49d2-955e-a4f1bb2a7f8e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9k8fv"
Jul 30 00:27:59.292687 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.292620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e048fda-d493-49d2-955e-a4f1bb2a7f8e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-9k8fv\" (UID: \"2e048fda-d493-49d2-955e-a4f1bb2a7f8e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9k8fv"
Jul 30 00:27:59.293148 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.293130    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc56b79-4db8-4489-a8c5-95e6349ce60d-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-m7n6j\" (UID: \"9fc56b79-4db8-4489-a8c5-95e6349ce60d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-m7n6j"
Jul 30 00:27:59.293293 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.293234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e048fda-d493-49d2-955e-a4f1bb2a7f8e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-9k8fv\" (UID: \"2e048fda-d493-49d2-955e-a4f1bb2a7f8e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9k8fv"
Jul 30 00:27:59.300459 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.300432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7s92\" (UniqueName: \"kubernetes.io/projected/2e048fda-d493-49d2-955e-a4f1bb2a7f8e-kube-api-access-c7s92\") pod \"payload-processing-5cc6cd444b-9k8fv\" (UID: \"2e048fda-d493-49d2-955e-a4f1bb2a7f8e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9k8fv"
Jul 30 00:27:59.300563 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.300482    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qdxn\" (UniqueName: \"kubernetes.io/projected/9fc56b79-4db8-4489-a8c5-95e6349ce60d-kube-api-access-5qdxn\") pod \"payload-pre-processing-687dbb6cf6-m7n6j\" (UID: \"9fc56b79-4db8-4489-a8c5-95e6349ce60d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-m7n6j"
Jul 30 00:27:59.420894 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.420870    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-m7n6j"
Jul 30 00:27:59.457929 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.457866    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-9k8fv"
Jul 30 00:27:59.563919 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.563890    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-m7n6j"]
Jul 30 00:27:59.565859 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:27:59.565828    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc56b79_4db8_4489_a8c5_95e6349ce60d.slice/crio-dbb28c79a752301ea44547dd376e97c161bf22ca0dd13de4c9c1d7e5006a03f8 WatchSource:0}: Error finding container dbb28c79a752301ea44547dd376e97c161bf22ca0dd13de4c9c1d7e5006a03f8: Status 404 returned error can't find the container with id dbb28c79a752301ea44547dd376e97c161bf22ca0dd13de4c9c1d7e5006a03f8
Jul 30 00:27:59.601471 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:27:59.601442    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-9k8fv"]
Jul 30 00:27:59.604869 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:27:59.604837    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e048fda_d493_49d2_955e_a4f1bb2a7f8e.slice/crio-436b77ae9da71d71076585691c23c3d35c96141b891ac0918a7efd06afdf1ade WatchSource:0}: Error finding container 436b77ae9da71d71076585691c23c3d35c96141b891ac0918a7efd06afdf1ade: Status 404 returned error can't find the container with id 436b77ae9da71d71076585691c23c3d35c96141b891ac0918a7efd06afdf1ade
Jul 30 00:28:00.048025 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:00.047991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68d9bb9c54-bh2mr" event={"ID":"57b0496d-ddb4-4f0f-8684-f8031487b466","Type":"ContainerStarted","Data":"1086412bfc521a84920ba9d4addeacd1319bec1c4bf473c2062255110bb3e32b"}
Jul 30 00:28:00.049088 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:00.049061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-9k8fv" event={"ID":"2e048fda-d493-49d2-955e-a4f1bb2a7f8e","Type":"ContainerStarted","Data":"436b77ae9da71d71076585691c23c3d35c96141b891ac0918a7efd06afdf1ade"}
Jul 30 00:28:00.050138 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:00.050121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-m7n6j" event={"ID":"9fc56b79-4db8-4489-a8c5-95e6349ce60d","Type":"ContainerStarted","Data":"dbb28c79a752301ea44547dd376e97c161bf22ca0dd13de4c9c1d7e5006a03f8"}
Jul 30 00:28:02.062175 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:02.062138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-9k8fv" event={"ID":"2e048fda-d493-49d2-955e-a4f1bb2a7f8e","Type":"ContainerStarted","Data":"f75314530a24f5f0690d4a60832d000e5238a2fcd1ffa8173f92909e1baaa3ba"}
Jul 30 00:28:02.062618 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:02.062276    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-9k8fv"
Jul 30 00:28:02.063864 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:02.063839    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-m7n6j" event={"ID":"9fc56b79-4db8-4489-a8c5-95e6349ce60d","Type":"ContainerStarted","Data":"72bb25c2b460662c69a1683ae1255c84980ca61d671d472940d31d62dc9eb1c2"}
Jul 30 00:28:02.063978 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:02.063958    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-m7n6j"
Jul 30 00:28:02.079999 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:02.079950    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-9k8fv" podStartSLOduration=1.379557882 podStartE2EDuration="3.079931859s" totalImagesPullingTime="1.700373977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:28:02.078273727 +0000 UTC m=+607.375540660" watchObservedRunningTime="2026-07-30 00:28:02.079931859 +0000 UTC m=+607.377198786"
Jul 30 00:28:02.093630 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:02.093583    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-m7n6j" podStartSLOduration=1.361149196 podStartE2EDuration="3.093569418s" totalImagesPullingTime="1.732420222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:28:02.092667598 +0000 UTC m=+607.389934532" watchObservedRunningTime="2026-07-30 00:28:02.093569418 +0000 UTC m=+607.390836349"
Jul 30 00:28:04.076872 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:04.076841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68d9bb9c54-bh2mr" event={"ID":"57b0496d-ddb4-4f0f-8684-f8031487b466","Type":"ContainerStarted","Data":"bcb2c4cf89315c68aaf59babeebeb8f6403f127e599d6390a0299baa5e3b71bb"}
Jul 30 00:28:04.077286 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:04.076942    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-68d9bb9c54-bh2mr"
Jul 30 00:28:04.094086 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:04.094042    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-68d9bb9c54-bh2mr" podStartSLOduration=2.111756906 podStartE2EDuration="6.094027027s" totalImagesPullingTime="3.982270121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:28:04.092343199 +0000 UTC m=+609.389610132" watchObservedRunningTime="2026-07-30 00:28:04.094027027 +0000 UTC m=+609.391293962"
Jul 30 00:28:10.086530 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:10.086504    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-68d9bb9c54-bh2mr"
Jul 30 00:28:13.073099 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:13.073065    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-m7n6j"
Jul 30 00:28:13.073514 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:13.073137    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-9k8fv"
Jul 30 00:28:43.903062 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:43.903031    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"]
Jul 30 00:28:43.906180 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:43.906160    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:43.910053 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:43.910034    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 30 00:28:43.910154 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:43.910060    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 30 00:28:43.910154 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:43.910139    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fwqxf\""
Jul 30 00:28:43.910272 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:43.910161    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 30 00:28:43.916520 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:43.916496    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"]
Jul 30 00:28:43.979089 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:43.979061    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nftqr\" (UID: \"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:43.979277 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:43.979125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nftqr\" (UID: \"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:43.979277 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:43.979144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp586\" (UniqueName: \"kubernetes.io/projected/ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b-kube-api-access-zp586\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nftqr\" (UID: \"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:43.979277 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:43.979165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nftqr\" (UID: \"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:43.979277 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:43.979181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nftqr\" (UID: \"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:43.979432 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:43.979286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nftqr\" (UID: \"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:44.080350 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:44.080323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nftqr\" (UID: \"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:44.080466 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:44.080366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nftqr\" (UID: \"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:44.080466 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:44.080412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nftqr\" (UID: \"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:44.080466 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:44.080430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp586\" (UniqueName: \"kubernetes.io/projected/ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b-kube-api-access-zp586\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nftqr\" (UID: \"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:44.080466 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:44.080449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nftqr\" (UID: \"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:44.080658 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:44.080473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nftqr\" (UID: \"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:44.080778 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:44.080758    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nftqr\" (UID: \"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:44.080917 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:44.080897    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nftqr\" (UID: \"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:44.080962 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:44.080912    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nftqr\" (UID: \"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:44.082720 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:44.082698    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nftqr\" (UID: \"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:44.082938 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:44.082920    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nftqr\" (UID: \"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:44.088128 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:44.088110    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp586\" (UniqueName: \"kubernetes.io/projected/ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b-kube-api-access-zp586\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nftqr\" (UID: \"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:44.216589 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:44.216522    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:44.354743 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:44.354712    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"]
Jul 30 00:28:44.358113 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:28:44.358080    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba605987_e2b2_4e1c_98c2_f8cfb8b6e17b.slice/crio-6de31e39fe2cd8a119e53f68104ae1bf0ead24d5b338405fa6186ea009b819ea WatchSource:0}: Error finding container 6de31e39fe2cd8a119e53f68104ae1bf0ead24d5b338405fa6186ea009b819ea: Status 404 returned error can't find the container with id 6de31e39fe2cd8a119e53f68104ae1bf0ead24d5b338405fa6186ea009b819ea
Jul 30 00:28:44.359444 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:44.359427    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 00:28:45.293834 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:45.293785    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr" event={"ID":"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b","Type":"ContainerStarted","Data":"6de31e39fe2cd8a119e53f68104ae1bf0ead24d5b338405fa6186ea009b819ea"}
Jul 30 00:28:46.300225 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:46.300184    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr" event={"ID":"ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b","Type":"ContainerStarted","Data":"8734b94320a0283788e236c296692e2540069f929abbae49899b2c632b7cc82b"}
Jul 30 00:28:46.317144 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:46.317099    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr" podStartSLOduration=2.286002868 podStartE2EDuration="3.317084478s" totalImagesPullingTime="1.03108161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:28:46.3169279 +0000 UTC m=+651.614194833" watchObservedRunningTime="2026-07-30 00:28:46.317084478 +0000 UTC m=+651.614351411"
Jul 30 00:28:54.216982 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:54.216946    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:54.217399 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:54.217022    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:54.229341 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:54.229311    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:54.348610 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:54.348585    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nftqr"
Jul 30 00:28:57.219381 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.219339    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"]
Jul 30 00:28:57.222204 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.222182    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.224722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.224703    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 30 00:28:57.235558 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.235533    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"]
Jul 30 00:28:57.296563 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.296524    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd1c313-29f1-4742-b3a7-d46a063bc081-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw\" (UID: \"afd1c313-29f1-4742-b3a7-d46a063bc081\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.296563 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.296565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afd1c313-29f1-4742-b3a7-d46a063bc081-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw\" (UID: \"afd1c313-29f1-4742-b3a7-d46a063bc081\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.296808 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.296593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afd1c313-29f1-4742-b3a7-d46a063bc081-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw\" (UID: \"afd1c313-29f1-4742-b3a7-d46a063bc081\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.296808 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.296663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2drzn\" (UniqueName: \"kubernetes.io/projected/afd1c313-29f1-4742-b3a7-d46a063bc081-kube-api-access-2drzn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw\" (UID: \"afd1c313-29f1-4742-b3a7-d46a063bc081\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.296808 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.296789    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afd1c313-29f1-4742-b3a7-d46a063bc081-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw\" (UID: \"afd1c313-29f1-4742-b3a7-d46a063bc081\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.296955 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.296840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afd1c313-29f1-4742-b3a7-d46a063bc081-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw\" (UID: \"afd1c313-29f1-4742-b3a7-d46a063bc081\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.398097 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.398066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afd1c313-29f1-4742-b3a7-d46a063bc081-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw\" (UID: \"afd1c313-29f1-4742-b3a7-d46a063bc081\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.398300 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.398134    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd1c313-29f1-4742-b3a7-d46a063bc081-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw\" (UID: \"afd1c313-29f1-4742-b3a7-d46a063bc081\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.398300 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.398154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afd1c313-29f1-4742-b3a7-d46a063bc081-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw\" (UID: \"afd1c313-29f1-4742-b3a7-d46a063bc081\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.398300 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.398172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afd1c313-29f1-4742-b3a7-d46a063bc081-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw\" (UID: \"afd1c313-29f1-4742-b3a7-d46a063bc081\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.398300 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.398193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2drzn\" (UniqueName: \"kubernetes.io/projected/afd1c313-29f1-4742-b3a7-d46a063bc081-kube-api-access-2drzn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw\" (UID: \"afd1c313-29f1-4742-b3a7-d46a063bc081\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.398300 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.398228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afd1c313-29f1-4742-b3a7-d46a063bc081-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw\" (UID: \"afd1c313-29f1-4742-b3a7-d46a063bc081\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.398586 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.398562    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afd1c313-29f1-4742-b3a7-d46a063bc081-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw\" (UID: \"afd1c313-29f1-4742-b3a7-d46a063bc081\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.398644 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.398625    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd1c313-29f1-4742-b3a7-d46a063bc081-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw\" (UID: \"afd1c313-29f1-4742-b3a7-d46a063bc081\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.398705 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.398653    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afd1c313-29f1-4742-b3a7-d46a063bc081-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw\" (UID: \"afd1c313-29f1-4742-b3a7-d46a063bc081\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.400590 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.400567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afd1c313-29f1-4742-b3a7-d46a063bc081-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw\" (UID: \"afd1c313-29f1-4742-b3a7-d46a063bc081\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.400858 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.400835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afd1c313-29f1-4742-b3a7-d46a063bc081-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw\" (UID: \"afd1c313-29f1-4742-b3a7-d46a063bc081\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.406481 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.406457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2drzn\" (UniqueName: \"kubernetes.io/projected/afd1c313-29f1-4742-b3a7-d46a063bc081-kube-api-access-2drzn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw\" (UID: \"afd1c313-29f1-4742-b3a7-d46a063bc081\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.531579 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.531490    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:28:57.673861 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:57.673838    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"]
Jul 30 00:28:57.676140 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:28:57.676108    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafd1c313_29f1_4742_b3a7_d46a063bc081.slice/crio-c795457ad772083af0649eb84a55495ec646be090e5e0d10e7e27a4699310af2 WatchSource:0}: Error finding container c795457ad772083af0649eb84a55495ec646be090e5e0d10e7e27a4699310af2: Status 404 returned error can't find the container with id c795457ad772083af0649eb84a55495ec646be090e5e0d10e7e27a4699310af2
Jul 30 00:28:58.356615 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:58.356528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw" event={"ID":"afd1c313-29f1-4742-b3a7-d46a063bc081","Type":"ContainerStarted","Data":"2dd4032db6ef60af22aff1d86d9681c9d5df5981ab580713c104201443fd7b62"}
Jul 30 00:28:58.356615 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:58.356572    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw" event={"ID":"afd1c313-29f1-4742-b3a7-d46a063bc081","Type":"ContainerStarted","Data":"c795457ad772083af0649eb84a55495ec646be090e5e0d10e7e27a4699310af2"}
Jul 30 00:28:58.374749 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:28:58.374694    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw" podStartSLOduration=1.173652051 podStartE2EDuration="1.37467643s" totalImagesPullingTime="201.024379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:28:58.373562058 +0000 UTC m=+663.670828990" watchObservedRunningTime="2026-07-30 00:28:58.37467643 +0000 UTC m=+663.671943363"
Jul 30 00:29:07.532360 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:29:07.532330    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:29:07.532735 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:29:07.532371    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:29:07.544986 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:29:07.544963    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:29:08.416408 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:29:08.416382    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw"
Jul 30 00:30:17.195296 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:30:17.195234    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7fddc5c74d-qcfng"]
Jul 30 00:30:17.195853 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:30:17.195491    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7fddc5c74d-qcfng" podUID="1837fa42-9342-468b-bc78-58900786eaf1" containerName="authorino" containerID="cri-o://87761dcc38b8ad033e39e0fa36475872f2e44d6ffb54cf92c5bb862e2062d486" gracePeriod=30
Jul 30 00:30:17.447122 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:30:17.447056    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fddc5c74d-qcfng"
Jul 30 00:30:17.492044 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:30:17.492013    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1837fa42-9342-468b-bc78-58900786eaf1-tls-cert\") pod \"1837fa42-9342-468b-bc78-58900786eaf1\" (UID: \"1837fa42-9342-468b-bc78-58900786eaf1\") "
Jul 30 00:30:17.492219 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:30:17.492111    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmgqx\" (UniqueName: \"kubernetes.io/projected/1837fa42-9342-468b-bc78-58900786eaf1-kube-api-access-tmgqx\") pod \"1837fa42-9342-468b-bc78-58900786eaf1\" (UID: \"1837fa42-9342-468b-bc78-58900786eaf1\") "
Jul 30 00:30:17.494083 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:30:17.494041    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1837fa42-9342-468b-bc78-58900786eaf1-kube-api-access-tmgqx" (OuterVolumeSpecName: "kube-api-access-tmgqx") pod "1837fa42-9342-468b-bc78-58900786eaf1" (UID: "1837fa42-9342-468b-bc78-58900786eaf1"). InnerVolumeSpecName "kube-api-access-tmgqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:30:17.501911 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:30:17.501879    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1837fa42-9342-468b-bc78-58900786eaf1-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "1837fa42-9342-468b-bc78-58900786eaf1" (UID: "1837fa42-9342-468b-bc78-58900786eaf1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:30:17.593050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:30:17.593023    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1837fa42-9342-468b-bc78-58900786eaf1-tls-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:30:17.593050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:30:17.593048    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmgqx\" (UniqueName: \"kubernetes.io/projected/1837fa42-9342-468b-bc78-58900786eaf1-kube-api-access-tmgqx\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:30:17.721226 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:30:17.721152    2579 generic.go:356] "Generic (PLEG): container finished" podID="1837fa42-9342-468b-bc78-58900786eaf1" containerID="87761dcc38b8ad033e39e0fa36475872f2e44d6ffb54cf92c5bb862e2062d486" exitCode=0
Jul 30 00:30:17.721226 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:30:17.721195    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fddc5c74d-qcfng"
Jul 30 00:30:17.721226 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:30:17.721203    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fddc5c74d-qcfng" event={"ID":"1837fa42-9342-468b-bc78-58900786eaf1","Type":"ContainerDied","Data":"87761dcc38b8ad033e39e0fa36475872f2e44d6ffb54cf92c5bb862e2062d486"}
Jul 30 00:30:17.721475 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:30:17.721232    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fddc5c74d-qcfng" event={"ID":"1837fa42-9342-468b-bc78-58900786eaf1","Type":"ContainerDied","Data":"e88e92fad0e4dd5eda59db144aef782baf432031e5d6ee2919f78292f0d91197"}
Jul 30 00:30:17.721475 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:30:17.721273    2579 scope.go:117] "RemoveContainer" containerID="87761dcc38b8ad033e39e0fa36475872f2e44d6ffb54cf92c5bb862e2062d486"
Jul 30 00:30:17.732089 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:30:17.732057    2579 scope.go:117] "RemoveContainer" containerID="87761dcc38b8ad033e39e0fa36475872f2e44d6ffb54cf92c5bb862e2062d486"
Jul 30 00:30:17.732446 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:30:17.732426    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87761dcc38b8ad033e39e0fa36475872f2e44d6ffb54cf92c5bb862e2062d486\": container with ID starting with 87761dcc38b8ad033e39e0fa36475872f2e44d6ffb54cf92c5bb862e2062d486 not found: ID does not exist" containerID="87761dcc38b8ad033e39e0fa36475872f2e44d6ffb54cf92c5bb862e2062d486"
Jul 30 00:30:17.732551 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:30:17.732452    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87761dcc38b8ad033e39e0fa36475872f2e44d6ffb54cf92c5bb862e2062d486"} err="failed to get container status \"87761dcc38b8ad033e39e0fa36475872f2e44d6ffb54cf92c5bb862e2062d486\": rpc error: code = NotFound desc = could not find container \"87761dcc38b8ad033e39e0fa36475872f2e44d6ffb54cf92c5bb862e2062d486\": container with ID starting with 87761dcc38b8ad033e39e0fa36475872f2e44d6ffb54cf92c5bb862e2062d486 not found: ID does not exist"
Jul 30 00:30:17.743852 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:30:17.743829    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7fddc5c74d-qcfng"]
Jul 30 00:30:17.747650 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:30:17.747632    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7fddc5c74d-qcfng"]
Jul 30 00:30:19.322708 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:30:19.322673    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1837fa42-9342-468b-bc78-58900786eaf1" path="/var/lib/kubelet/pods/1837fa42-9342-468b-bc78-58900786eaf1/volumes"
Jul 30 00:32:08.216581 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:08.216545    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bd495bbc5-lfvb5"]
Jul 30 00:32:08.216991 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:08.216930    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1837fa42-9342-468b-bc78-58900786eaf1" containerName="authorino"
Jul 30 00:32:08.216991 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:08.216942    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1837fa42-9342-468b-bc78-58900786eaf1" containerName="authorino"
Jul 30 00:32:08.217062 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:08.217007    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1837fa42-9342-468b-bc78-58900786eaf1" containerName="authorino"
Jul 30 00:32:08.219911 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:08.219896    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd495bbc5-lfvb5"
Jul 30 00:32:08.223927 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:08.223893    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 30 00:32:08.224065 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:08.223957    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-94z2g\""
Jul 30 00:32:08.224065 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:08.224027    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 30 00:32:08.229269 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:08.229228    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bd495bbc5-lfvb5"]
Jul 30 00:32:08.342154 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:08.342128    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0b5c710-7002-4e32-92e3-97d243da28b2-webhook-cert\") pod \"maas-controller-bd495bbc5-lfvb5\" (UID: \"f0b5c710-7002-4e32-92e3-97d243da28b2\") " pod="opendatahub/maas-controller-bd495bbc5-lfvb5"
Jul 30 00:32:08.342311 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:08.342282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xhph\" (UniqueName: \"kubernetes.io/projected/f0b5c710-7002-4e32-92e3-97d243da28b2-kube-api-access-5xhph\") pod \"maas-controller-bd495bbc5-lfvb5\" (UID: \"f0b5c710-7002-4e32-92e3-97d243da28b2\") " pod="opendatahub/maas-controller-bd495bbc5-lfvb5"
Jul 30 00:32:08.443275 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:08.443224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xhph\" (UniqueName: \"kubernetes.io/projected/f0b5c710-7002-4e32-92e3-97d243da28b2-kube-api-access-5xhph\") pod \"maas-controller-bd495bbc5-lfvb5\" (UID: \"f0b5c710-7002-4e32-92e3-97d243da28b2\") " pod="opendatahub/maas-controller-bd495bbc5-lfvb5"
Jul 30 00:32:08.443432 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:08.443293    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0b5c710-7002-4e32-92e3-97d243da28b2-webhook-cert\") pod \"maas-controller-bd495bbc5-lfvb5\" (UID: \"f0b5c710-7002-4e32-92e3-97d243da28b2\") " pod="opendatahub/maas-controller-bd495bbc5-lfvb5"
Jul 30 00:32:08.445655 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:08.445633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0b5c710-7002-4e32-92e3-97d243da28b2-webhook-cert\") pod \"maas-controller-bd495bbc5-lfvb5\" (UID: \"f0b5c710-7002-4e32-92e3-97d243da28b2\") " pod="opendatahub/maas-controller-bd495bbc5-lfvb5"
Jul 30 00:32:08.451485 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:08.451469    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xhph\" (UniqueName: \"kubernetes.io/projected/f0b5c710-7002-4e32-92e3-97d243da28b2-kube-api-access-5xhph\") pod \"maas-controller-bd495bbc5-lfvb5\" (UID: \"f0b5c710-7002-4e32-92e3-97d243da28b2\") " pod="opendatahub/maas-controller-bd495bbc5-lfvb5"
Jul 30 00:32:08.530875 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:08.530810    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd495bbc5-lfvb5"
Jul 30 00:32:08.660206 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:08.660181    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bd495bbc5-lfvb5"]
Jul 30 00:32:08.662346 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:32:08.662319    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b5c710_7002_4e32_92e3_97d243da28b2.slice/crio-f0f8bff3fd92d278a25a081eb6c7ed5c010023eda553393f4ded353c3a225c98 WatchSource:0}: Error finding container f0f8bff3fd92d278a25a081eb6c7ed5c010023eda553393f4ded353c3a225c98: Status 404 returned error can't find the container with id f0f8bff3fd92d278a25a081eb6c7ed5c010023eda553393f4ded353c3a225c98
Jul 30 00:32:09.244649 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:09.244615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd495bbc5-lfvb5" event={"ID":"f0b5c710-7002-4e32-92e3-97d243da28b2","Type":"ContainerStarted","Data":"f0f8bff3fd92d278a25a081eb6c7ed5c010023eda553393f4ded353c3a225c98"}
Jul 30 00:32:12.260835 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:12.260795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd495bbc5-lfvb5" event={"ID":"f0b5c710-7002-4e32-92e3-97d243da28b2","Type":"ContainerStarted","Data":"18c1ed3ac88e511df4050ba4f31faa93ff03f0a8a4643300734ee22fbb879948"}
Jul 30 00:32:12.261256 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:12.260885    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bd495bbc5-lfvb5"
Jul 30 00:32:12.280105 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:12.280049    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bd495bbc5-lfvb5" podStartSLOduration=1.624673981 podStartE2EDuration="4.280034388s" totalImagesPullingTime="2.655360407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:32:12.277277237 +0000 UTC m=+857.574544168" watchObservedRunningTime="2026-07-30 00:32:12.280034388 +0000 UTC m=+857.577301320"
Jul 30 00:32:23.269425 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:23.269388    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bd495bbc5-lfvb5"
Jul 30 00:32:27.890478 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:27.890441    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bd495bbc5-lfvb5"]
Jul 30 00:32:27.890875 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:27.890644    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-bd495bbc5-lfvb5" podUID="f0b5c710-7002-4e32-92e3-97d243da28b2" containerName="manager" containerID="cri-o://18c1ed3ac88e511df4050ba4f31faa93ff03f0a8a4643300734ee22fbb879948" gracePeriod=30
Jul 30 00:32:28.137457 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:28.137434    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd495bbc5-lfvb5"
Jul 30 00:32:28.215482 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:28.215405    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0b5c710-7002-4e32-92e3-97d243da28b2-webhook-cert\") pod \"f0b5c710-7002-4e32-92e3-97d243da28b2\" (UID: \"f0b5c710-7002-4e32-92e3-97d243da28b2\") "
Jul 30 00:32:28.215619 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:28.215485    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xhph\" (UniqueName: \"kubernetes.io/projected/f0b5c710-7002-4e32-92e3-97d243da28b2-kube-api-access-5xhph\") pod \"f0b5c710-7002-4e32-92e3-97d243da28b2\" (UID: \"f0b5c710-7002-4e32-92e3-97d243da28b2\") "
Jul 30 00:32:28.217471 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:28.217446    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0b5c710-7002-4e32-92e3-97d243da28b2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f0b5c710-7002-4e32-92e3-97d243da28b2" (UID: "f0b5c710-7002-4e32-92e3-97d243da28b2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:32:28.217548 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:28.217447    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b5c710-7002-4e32-92e3-97d243da28b2-kube-api-access-5xhph" (OuterVolumeSpecName: "kube-api-access-5xhph") pod "f0b5c710-7002-4e32-92e3-97d243da28b2" (UID: "f0b5c710-7002-4e32-92e3-97d243da28b2"). InnerVolumeSpecName "kube-api-access-5xhph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:32:28.316229 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:28.316198    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0b5c710-7002-4e32-92e3-97d243da28b2-webhook-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:32:28.316229 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:28.316224    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xhph\" (UniqueName: \"kubernetes.io/projected/f0b5c710-7002-4e32-92e3-97d243da28b2-kube-api-access-5xhph\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:32:28.337061 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:28.337031    2579 generic.go:356] "Generic (PLEG): container finished" podID="f0b5c710-7002-4e32-92e3-97d243da28b2" containerID="18c1ed3ac88e511df4050ba4f31faa93ff03f0a8a4643300734ee22fbb879948" exitCode=0
Jul 30 00:32:28.337204 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:28.337073    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd495bbc5-lfvb5"
Jul 30 00:32:28.337204 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:28.337078    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd495bbc5-lfvb5" event={"ID":"f0b5c710-7002-4e32-92e3-97d243da28b2","Type":"ContainerDied","Data":"18c1ed3ac88e511df4050ba4f31faa93ff03f0a8a4643300734ee22fbb879948"}
Jul 30 00:32:28.337204 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:28.337119    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd495bbc5-lfvb5" event={"ID":"f0b5c710-7002-4e32-92e3-97d243da28b2","Type":"ContainerDied","Data":"f0f8bff3fd92d278a25a081eb6c7ed5c010023eda553393f4ded353c3a225c98"}
Jul 30 00:32:28.337204 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:28.337138    2579 scope.go:117] "RemoveContainer" containerID="18c1ed3ac88e511df4050ba4f31faa93ff03f0a8a4643300734ee22fbb879948"
Jul 30 00:32:28.346062 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:28.346042    2579 scope.go:117] "RemoveContainer" containerID="18c1ed3ac88e511df4050ba4f31faa93ff03f0a8a4643300734ee22fbb879948"
Jul 30 00:32:28.346347 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:32:28.346325    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c1ed3ac88e511df4050ba4f31faa93ff03f0a8a4643300734ee22fbb879948\": container with ID starting with 18c1ed3ac88e511df4050ba4f31faa93ff03f0a8a4643300734ee22fbb879948 not found: ID does not exist" containerID="18c1ed3ac88e511df4050ba4f31faa93ff03f0a8a4643300734ee22fbb879948"
Jul 30 00:32:28.346428 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:28.346354    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c1ed3ac88e511df4050ba4f31faa93ff03f0a8a4643300734ee22fbb879948"} err="failed to get container status \"18c1ed3ac88e511df4050ba4f31faa93ff03f0a8a4643300734ee22fbb879948\": rpc error: code = NotFound desc = could not find container \"18c1ed3ac88e511df4050ba4f31faa93ff03f0a8a4643300734ee22fbb879948\": container with ID starting with 18c1ed3ac88e511df4050ba4f31faa93ff03f0a8a4643300734ee22fbb879948 not found: ID does not exist"
Jul 30 00:32:28.357858 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:28.357837    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bd495bbc5-lfvb5"]
Jul 30 00:32:28.361092 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:28.361072    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bd495bbc5-lfvb5"]
Jul 30 00:32:29.086263 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:29.086211    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bd495bbc5-jvmkl"]
Jul 30 00:32:29.086774 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:29.086754    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0b5c710-7002-4e32-92e3-97d243da28b2" containerName="manager"
Jul 30 00:32:29.086862 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:29.086777    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0b5c710-7002-4e32-92e3-97d243da28b2" containerName="manager"
Jul 30 00:32:29.086916 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:29.086869    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0b5c710-7002-4e32-92e3-97d243da28b2" containerName="manager"
Jul 30 00:32:29.090946 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:29.090924    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd495bbc5-jvmkl"
Jul 30 00:32:29.094927 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:29.094897    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 30 00:32:29.095049 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:29.094897    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-94z2g\""
Jul 30 00:32:29.095049 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:29.094904    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 30 00:32:29.097226 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:29.097200    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bd495bbc5-jvmkl"]
Jul 30 00:32:29.121859 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:29.121837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6440c0bc-07f7-4dfd-af29-c014028ecd2a-webhook-cert\") pod \"maas-controller-bd495bbc5-jvmkl\" (UID: \"6440c0bc-07f7-4dfd-af29-c014028ecd2a\") " pod="opendatahub/maas-controller-bd495bbc5-jvmkl"
Jul 30 00:32:29.121977 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:29.121876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbcw2\" (UniqueName: \"kubernetes.io/projected/6440c0bc-07f7-4dfd-af29-c014028ecd2a-kube-api-access-wbcw2\") pod \"maas-controller-bd495bbc5-jvmkl\" (UID: \"6440c0bc-07f7-4dfd-af29-c014028ecd2a\") " pod="opendatahub/maas-controller-bd495bbc5-jvmkl"
Jul 30 00:32:29.222914 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:29.222878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6440c0bc-07f7-4dfd-af29-c014028ecd2a-webhook-cert\") pod \"maas-controller-bd495bbc5-jvmkl\" (UID: \"6440c0bc-07f7-4dfd-af29-c014028ecd2a\") " pod="opendatahub/maas-controller-bd495bbc5-jvmkl"
Jul 30 00:32:29.223084 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:29.222926    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbcw2\" (UniqueName: \"kubernetes.io/projected/6440c0bc-07f7-4dfd-af29-c014028ecd2a-kube-api-access-wbcw2\") pod \"maas-controller-bd495bbc5-jvmkl\" (UID: \"6440c0bc-07f7-4dfd-af29-c014028ecd2a\") " pod="opendatahub/maas-controller-bd495bbc5-jvmkl"
Jul 30 00:32:29.225213 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:29.225191    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6440c0bc-07f7-4dfd-af29-c014028ecd2a-webhook-cert\") pod \"maas-controller-bd495bbc5-jvmkl\" (UID: \"6440c0bc-07f7-4dfd-af29-c014028ecd2a\") " pod="opendatahub/maas-controller-bd495bbc5-jvmkl"
Jul 30 00:32:29.231508 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:29.231490    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbcw2\" (UniqueName: \"kubernetes.io/projected/6440c0bc-07f7-4dfd-af29-c014028ecd2a-kube-api-access-wbcw2\") pod \"maas-controller-bd495bbc5-jvmkl\" (UID: \"6440c0bc-07f7-4dfd-af29-c014028ecd2a\") " pod="opendatahub/maas-controller-bd495bbc5-jvmkl"
Jul 30 00:32:29.327948 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:29.327921    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0b5c710-7002-4e32-92e3-97d243da28b2" path="/var/lib/kubelet/pods/f0b5c710-7002-4e32-92e3-97d243da28b2/volumes"
Jul 30 00:32:29.402412 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:29.402385    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd495bbc5-jvmkl"
Jul 30 00:32:29.530781 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:29.530756    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bd495bbc5-jvmkl"]
Jul 30 00:32:29.532558 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:32:29.532534    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6440c0bc_07f7_4dfd_af29_c014028ecd2a.slice/crio-57e08a3299bd5f0033bae35a9b62d25dcf0e79c0b96300bde9a56d010e4dbfc6 WatchSource:0}: Error finding container 57e08a3299bd5f0033bae35a9b62d25dcf0e79c0b96300bde9a56d010e4dbfc6: Status 404 returned error can't find the container with id 57e08a3299bd5f0033bae35a9b62d25dcf0e79c0b96300bde9a56d010e4dbfc6
Jul 30 00:32:30.349057 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:30.349021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd495bbc5-jvmkl" event={"ID":"6440c0bc-07f7-4dfd-af29-c014028ecd2a","Type":"ContainerStarted","Data":"57e08a3299bd5f0033bae35a9b62d25dcf0e79c0b96300bde9a56d010e4dbfc6"}
Jul 30 00:32:31.359214 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:31.359174    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd495bbc5-jvmkl" event={"ID":"6440c0bc-07f7-4dfd-af29-c014028ecd2a","Type":"ContainerStarted","Data":"a8c40007807897ba4a82052d572a87ffd519ec7d9efb4527283b7922b7a79cae"}
Jul 30 00:32:31.359641 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:31.359291    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bd495bbc5-jvmkl"
Jul 30 00:32:31.379188 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:31.379132    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bd495bbc5-jvmkl" podStartSLOduration=0.716415505 podStartE2EDuration="2.379116292s" totalImagesPullingTime="1.662700787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:32:31.376682696 +0000 UTC m=+876.673949629" watchObservedRunningTime="2026-07-30 00:32:31.379116292 +0000 UTC m=+876.676383225"
Jul 30 00:32:42.366568 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:42.366535    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bd495bbc5-jvmkl"
Jul 30 00:32:55.188176 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:55.188090    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:32:55.192142 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:32:55.192114    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:37:55.221979 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:37:55.221954    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:37:55.222497 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:37:55.221954    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:41:25.018494 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.018455    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"]
Jul 30 00:41:25.022011 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.021990    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.024761 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.024741    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-1ce7bab4-kserve-self-signed-certs\""
Jul 30 00:41:25.036850 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.036821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-home\") pod \"e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") " pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.036948 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.036858    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-model-cache\") pod \"e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") " pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.036948 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.036882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-tmp-dir\") pod \"e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") " pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.036948 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.036906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjxwk\" (UniqueName: \"kubernetes.io/projected/4ecb1f1f-c149-4269-b9d0-e642e8f01853-kube-api-access-qjxwk\") pod \"e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") " pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.037058 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.036961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ecb1f1f-c149-4269-b9d0-e642e8f01853-tls-certs\") pod \"e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") " pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.037058 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.037006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-dshm\") pod \"e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") " pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.039288 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.039268    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"]
Jul 30 00:41:25.138375 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.138349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-home\") pod \"e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") " pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.138375 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.138377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-model-cache\") pod \"e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") " pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.138637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.138397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-tmp-dir\") pod \"e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") " pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.138637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.138413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjxwk\" (UniqueName: \"kubernetes.io/projected/4ecb1f1f-c149-4269-b9d0-e642e8f01853-kube-api-access-qjxwk\") pod \"e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") " pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.138637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.138435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ecb1f1f-c149-4269-b9d0-e642e8f01853-tls-certs\") pod \"e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") " pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.138637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.138469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-dshm\") pod \"e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") " pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.138873 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.138791    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-tmp-dir\") pod \"e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") " pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.138873 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.138805    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-model-cache\") pod \"e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") " pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.138950 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.138918    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-home\") pod \"e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") " pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.140768 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.140745    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-dshm\") pod \"e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") " pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.140955 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.140935    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ecb1f1f-c149-4269-b9d0-e642e8f01853-tls-certs\") pod \"e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") " pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.146204 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.146181    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjxwk\" (UniqueName: \"kubernetes.io/projected/4ecb1f1f-c149-4269-b9d0-e642e8f01853-kube-api-access-qjxwk\") pod \"e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") " pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.332953 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.332887    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:25.464158 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.464130    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"]
Jul 30 00:41:25.466371 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:41:25.466343    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ecb1f1f_c149_4269_b9d0_e642e8f01853.slice/crio-375f12caacabbfa516ae8f93e615c005306eaaa80667411502caa0948d22786c WatchSource:0}: Error finding container 375f12caacabbfa516ae8f93e615c005306eaaa80667411502caa0948d22786c: Status 404 returned error can't find the container with id 375f12caacabbfa516ae8f93e615c005306eaaa80667411502caa0948d22786c
Jul 30 00:41:25.833204 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.833171    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls" event={"ID":"4ecb1f1f-c149-4269-b9d0-e642e8f01853","Type":"ContainerStarted","Data":"c92452396041f00447ad41891a5a0cb1785dc9bc81e46745f3ddd4adf539ba8b"}
Jul 30 00:41:25.833204 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.833216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls" event={"ID":"4ecb1f1f-c149-4269-b9d0-e642e8f01853","Type":"ContainerStarted","Data":"375f12caacabbfa516ae8f93e615c005306eaaa80667411502caa0948d22786c"}
Jul 30 00:41:25.849663 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:25.849618    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls" podStartSLOduration=1.849600742 podStartE2EDuration="1.849600742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:41:25.84782136 +0000 UTC m=+1411.145088293" watchObservedRunningTime="2026-07-30 00:41:25.849600742 +0000 UTC m=+1411.146867675"
Jul 30 00:41:32.620306 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.620276    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"]
Jul 30 00:41:32.624057 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.624040    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:32.626672 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.626642    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-1ce7bab4-kserve-self-signed-certs\""
Jul 30 00:41:32.633583 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.633552    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"]
Jul 30 00:41:32.708465 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.708406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-model-cache\") pod \"e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") " pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:32.708668 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.708475    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-home\") pod \"e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") " pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:32.708668 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.708500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-dshm\") pod \"e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") " pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:32.708668 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.708516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-tmp-dir\") pod \"e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") " pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:32.708668 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.708532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cxtn\" (UniqueName: \"kubernetes.io/projected/da9d25f0-e77a-46e2-b08f-b596242978d1-kube-api-access-9cxtn\") pod \"e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") " pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:32.708668 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.708631    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9d25f0-e77a-46e2-b08f-b596242978d1-tls-certs\") pod \"e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") " pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:32.809830 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.809800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-model-cache\") pod \"e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") " pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:32.809943 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.809835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-home\") pod \"e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") " pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:32.809943 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.809851    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-dshm\") pod \"e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") " pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:32.809943 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.809868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-tmp-dir\") pod \"e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") " pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:32.809943 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.809891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cxtn\" (UniqueName: \"kubernetes.io/projected/da9d25f0-e77a-46e2-b08f-b596242978d1-kube-api-access-9cxtn\") pod \"e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") " pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:32.809943 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.809920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9d25f0-e77a-46e2-b08f-b596242978d1-tls-certs\") pod \"e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") " pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:32.810304 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.810285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-model-cache\") pod \"e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") " pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:32.810373 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.810346    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-home\") pod \"e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") " pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:32.810434 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.810399    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-tmp-dir\") pod \"e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") " pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:32.812215 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.812194    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-dshm\") pod \"e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") " pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:32.812408 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.812390    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9d25f0-e77a-46e2-b08f-b596242978d1-tls-certs\") pod \"e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") " pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:32.826918 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.826896    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cxtn\" (UniqueName: \"kubernetes.io/projected/da9d25f0-e77a-46e2-b08f-b596242978d1-kube-api-access-9cxtn\") pod \"e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") " pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:32.934381 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:32.934317    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:33.072174 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:33.072150    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"]
Jul 30 00:41:33.074129 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:41:33.074094    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda9d25f0_e77a_46e2_b08f_b596242978d1.slice/crio-68c6dcb2e9585cc7d9c07ff1c1ad331da215d2ce1f7388b5e242fd9782847761 WatchSource:0}: Error finding container 68c6dcb2e9585cc7d9c07ff1c1ad331da215d2ce1f7388b5e242fd9782847761: Status 404 returned error can't find the container with id 68c6dcb2e9585cc7d9c07ff1c1ad331da215d2ce1f7388b5e242fd9782847761
Jul 30 00:41:33.873397 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:33.873366    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb" event={"ID":"da9d25f0-e77a-46e2-b08f-b596242978d1","Type":"ContainerStarted","Data":"e4e9e6d322b8bcfc20c0bf6e91470459fba3a488c2713cf4d191e26911bc91ad"}
Jul 30 00:41:33.873773 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:33.873407    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb" event={"ID":"da9d25f0-e77a-46e2-b08f-b596242978d1","Type":"ContainerStarted","Data":"68c6dcb2e9585cc7d9c07ff1c1ad331da215d2ce1f7388b5e242fd9782847761"}
Jul 30 00:41:33.891065 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:33.890996    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb" podStartSLOduration=1.890979979 podStartE2EDuration="1.890979979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:41:33.889148407 +0000 UTC m=+1419.186415339" watchObservedRunningTime="2026-07-30 00:41:33.890979979 +0000 UTC m=+1419.188246911"
Jul 30 00:41:35.333781 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:35.333701    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:35.333781 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:35.333748    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:35.346493 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:35.346471    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:35.895628 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:35.895601    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:41:42.934763 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:42.934735    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:42.935176 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:42.934775    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:42.946821 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:42.946794    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:43.933052 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:43.933014    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:41:55.421201 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:55.421120    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"]
Jul 30 00:41:56.799112 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:41:56.799080    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"]
Jul 30 00:42:10.441669 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.441628    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls" podUID="4ecb1f1f-c149-4269-b9d0-e642e8f01853" containerName="main" containerID="cri-o://c92452396041f00447ad41891a5a0cb1785dc9bc81e46745f3ddd4adf539ba8b" gracePeriod=45
Jul 30 00:42:10.692559 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.692500    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:42:10.746566 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.746540    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-model-cache\") pod \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") "
Jul 30 00:42:10.746705 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.746587    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjxwk\" (UniqueName: \"kubernetes.io/projected/4ecb1f1f-c149-4269-b9d0-e642e8f01853-kube-api-access-qjxwk\") pod \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") "
Jul 30 00:42:10.746705 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.746631    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-home\") pod \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") "
Jul 30 00:42:10.746705 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.746679    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-dshm\") pod \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") "
Jul 30 00:42:10.746824 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.746730    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-tmp-dir\") pod \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") "
Jul 30 00:42:10.746824 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.746754    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ecb1f1f-c149-4269-b9d0-e642e8f01853-tls-certs\") pod \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\" (UID: \"4ecb1f1f-c149-4269-b9d0-e642e8f01853\") "
Jul 30 00:42:10.746824 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.746784    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ecb1f1f-c149-4269-b9d0-e642e8f01853" (UID: "4ecb1f1f-c149-4269-b9d0-e642e8f01853"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:42:10.746969 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.746947    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-home" (OuterVolumeSpecName: "home") pod "4ecb1f1f-c149-4269-b9d0-e642e8f01853" (UID: "4ecb1f1f-c149-4269-b9d0-e642e8f01853"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:42:10.747030 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.746956    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ecb1f1f-c149-4269-b9d0-e642e8f01853" (UID: "4ecb1f1f-c149-4269-b9d0-e642e8f01853"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:42:10.747100 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.747082    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:42:10.747159 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.747102    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:42:10.747159 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.747113    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:42:10.748885 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.748849    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ecb1f1f-c149-4269-b9d0-e642e8f01853-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ecb1f1f-c149-4269-b9d0-e642e8f01853" (UID: "4ecb1f1f-c149-4269-b9d0-e642e8f01853"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:42:10.748992 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.748910    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-dshm" (OuterVolumeSpecName: "dshm") pod "4ecb1f1f-c149-4269-b9d0-e642e8f01853" (UID: "4ecb1f1f-c149-4269-b9d0-e642e8f01853"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:42:10.748992 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.748913    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ecb1f1f-c149-4269-b9d0-e642e8f01853-kube-api-access-qjxwk" (OuterVolumeSpecName: "kube-api-access-qjxwk") pod "4ecb1f1f-c149-4269-b9d0-e642e8f01853" (UID: "4ecb1f1f-c149-4269-b9d0-e642e8f01853"). InnerVolumeSpecName "kube-api-access-qjxwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:42:10.847539 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.847515    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ecb1f1f-c149-4269-b9d0-e642e8f01853-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:42:10.847539 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.847535    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ecb1f1f-c149-4269-b9d0-e642e8f01853-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:42:10.847701 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:10.847545    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjxwk\" (UniqueName: \"kubernetes.io/projected/4ecb1f1f-c149-4269-b9d0-e642e8f01853-kube-api-access-qjxwk\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:42:11.045162 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:11.045083    2579 generic.go:356] "Generic (PLEG): container finished" podID="4ecb1f1f-c149-4269-b9d0-e642e8f01853" containerID="c92452396041f00447ad41891a5a0cb1785dc9bc81e46745f3ddd4adf539ba8b" exitCode=0
Jul 30 00:42:11.045162 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:11.045131    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"
Jul 30 00:42:11.045399 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:11.045172    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls" event={"ID":"4ecb1f1f-c149-4269-b9d0-e642e8f01853","Type":"ContainerDied","Data":"c92452396041f00447ad41891a5a0cb1785dc9bc81e46745f3ddd4adf539ba8b"}
Jul 30 00:42:11.045399 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:11.045221    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls" event={"ID":"4ecb1f1f-c149-4269-b9d0-e642e8f01853","Type":"ContainerDied","Data":"375f12caacabbfa516ae8f93e615c005306eaaa80667411502caa0948d22786c"}
Jul 30 00:42:11.045399 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:11.045265    2579 scope.go:117] "RemoveContainer" containerID="c92452396041f00447ad41891a5a0cb1785dc9bc81e46745f3ddd4adf539ba8b"
Jul 30 00:42:11.054947 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:11.054929    2579 scope.go:117] "RemoveContainer" containerID="c92452396041f00447ad41891a5a0cb1785dc9bc81e46745f3ddd4adf539ba8b"
Jul 30 00:42:11.055193 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:42:11.055173    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c92452396041f00447ad41891a5a0cb1785dc9bc81e46745f3ddd4adf539ba8b\": container with ID starting with c92452396041f00447ad41891a5a0cb1785dc9bc81e46745f3ddd4adf539ba8b not found: ID does not exist" containerID="c92452396041f00447ad41891a5a0cb1785dc9bc81e46745f3ddd4adf539ba8b"
Jul 30 00:42:11.055294 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:11.055200    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c92452396041f00447ad41891a5a0cb1785dc9bc81e46745f3ddd4adf539ba8b"} err="failed to get container status \"c92452396041f00447ad41891a5a0cb1785dc9bc81e46745f3ddd4adf539ba8b\": rpc error: code = NotFound desc = could not find container \"c92452396041f00447ad41891a5a0cb1785dc9bc81e46745f3ddd4adf539ba8b\": container with ID starting with c92452396041f00447ad41891a5a0cb1785dc9bc81e46745f3ddd4adf539ba8b not found: ID does not exist"
Jul 30 00:42:11.067013 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:11.066990    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"]
Jul 30 00:42:11.070298 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:11.070279    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-1ce7bab4-kserve-8b74b8f9f-rhbls"]
Jul 30 00:42:11.322802 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:11.322740    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ecb1f1f-c149-4269-b9d0-e642e8f01853" path="/var/lib/kubelet/pods/4ecb1f1f-c149-4269-b9d0-e642e8f01853/volumes"
Jul 30 00:42:11.818638 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:11.818592    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb" podUID="da9d25f0-e77a-46e2-b08f-b596242978d1" containerName="main" containerID="cri-o://e4e9e6d322b8bcfc20c0bf6e91470459fba3a488c2713cf4d191e26911bc91ad" gracePeriod=45
Jul 30 00:42:12.053290 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.053214    2579 generic.go:356] "Generic (PLEG): container finished" podID="da9d25f0-e77a-46e2-b08f-b596242978d1" containerID="e4e9e6d322b8bcfc20c0bf6e91470459fba3a488c2713cf4d191e26911bc91ad" exitCode=0
Jul 30 00:42:12.053415 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.053293    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb" event={"ID":"da9d25f0-e77a-46e2-b08f-b596242978d1","Type":"ContainerDied","Data":"e4e9e6d322b8bcfc20c0bf6e91470459fba3a488c2713cf4d191e26911bc91ad"}
Jul 30 00:42:12.073446 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.073388    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:42:12.156138 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.156112    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-model-cache\") pod \"da9d25f0-e77a-46e2-b08f-b596242978d1\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") "
Jul 30 00:42:12.156309 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.156147    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-home\") pod \"da9d25f0-e77a-46e2-b08f-b596242978d1\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") "
Jul 30 00:42:12.156309 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.156167    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cxtn\" (UniqueName: \"kubernetes.io/projected/da9d25f0-e77a-46e2-b08f-b596242978d1-kube-api-access-9cxtn\") pod \"da9d25f0-e77a-46e2-b08f-b596242978d1\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") "
Jul 30 00:42:12.156309 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.156195    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-dshm\") pod \"da9d25f0-e77a-46e2-b08f-b596242978d1\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") "
Jul 30 00:42:12.156309 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.156211    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9d25f0-e77a-46e2-b08f-b596242978d1-tls-certs\") pod \"da9d25f0-e77a-46e2-b08f-b596242978d1\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") "
Jul 30 00:42:12.156309 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.156278    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-tmp-dir\") pod \"da9d25f0-e77a-46e2-b08f-b596242978d1\" (UID: \"da9d25f0-e77a-46e2-b08f-b596242978d1\") "
Jul 30 00:42:12.156531 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.156416    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "da9d25f0-e77a-46e2-b08f-b596242978d1" (UID: "da9d25f0-e77a-46e2-b08f-b596242978d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:42:12.156531 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.156474    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-home" (OuterVolumeSpecName: "home") pod "da9d25f0-e77a-46e2-b08f-b596242978d1" (UID: "da9d25f0-e77a-46e2-b08f-b596242978d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:42:12.156643 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.156605    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:42:12.156690 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.156647    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:42:12.156690 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.156642    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da9d25f0-e77a-46e2-b08f-b596242978d1" (UID: "da9d25f0-e77a-46e2-b08f-b596242978d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:42:12.158432 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.158406    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9d25f0-e77a-46e2-b08f-b596242978d1-kube-api-access-9cxtn" (OuterVolumeSpecName: "kube-api-access-9cxtn") pod "da9d25f0-e77a-46e2-b08f-b596242978d1" (UID: "da9d25f0-e77a-46e2-b08f-b596242978d1"). InnerVolumeSpecName "kube-api-access-9cxtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:42:12.158510 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.158443    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-dshm" (OuterVolumeSpecName: "dshm") pod "da9d25f0-e77a-46e2-b08f-b596242978d1" (UID: "da9d25f0-e77a-46e2-b08f-b596242978d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:42:12.158510 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.158464    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da9d25f0-e77a-46e2-b08f-b596242978d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da9d25f0-e77a-46e2-b08f-b596242978d1" (UID: "da9d25f0-e77a-46e2-b08f-b596242978d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:42:12.258092 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.258067    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cxtn\" (UniqueName: \"kubernetes.io/projected/da9d25f0-e77a-46e2-b08f-b596242978d1-kube-api-access-9cxtn\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:42:12.258092 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.258090    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:42:12.258267 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.258108    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9d25f0-e77a-46e2-b08f-b596242978d1-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:42:12.258267 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:12.258117    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9d25f0-e77a-46e2-b08f-b596242978d1-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:42:13.059596 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:13.059571    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"
Jul 30 00:42:13.059980 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:13.059617    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb" event={"ID":"da9d25f0-e77a-46e2-b08f-b596242978d1","Type":"ContainerDied","Data":"68c6dcb2e9585cc7d9c07ff1c1ad331da215d2ce1f7388b5e242fd9782847761"}
Jul 30 00:42:13.059980 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:13.059667    2579 scope.go:117] "RemoveContainer" containerID="e4e9e6d322b8bcfc20c0bf6e91470459fba3a488c2713cf4d191e26911bc91ad"
Jul 30 00:42:13.080469 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:13.080446    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"]
Jul 30 00:42:13.083469 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:13.083447    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-1ce7bab4-kserve-768c447b75-lvzlb"]
Jul 30 00:42:13.322499 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:13.322423    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da9d25f0-e77a-46e2-b08f-b596242978d1" path="/var/lib/kubelet/pods/da9d25f0-e77a-46e2-b08f-b596242978d1/volumes"
Jul 30 00:42:55.251640 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:55.251606    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:42:55.252380 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:42:55.252362    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:45:00.143342 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:00.143264    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756205-pr8xl"]
Jul 30 00:45:00.143800 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:00.143639    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ecb1f1f-c149-4269-b9d0-e642e8f01853" containerName="main"
Jul 30 00:45:00.143800 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:00.143650    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ecb1f1f-c149-4269-b9d0-e642e8f01853" containerName="main"
Jul 30 00:45:00.143800 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:00.143663    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da9d25f0-e77a-46e2-b08f-b596242978d1" containerName="main"
Jul 30 00:45:00.143800 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:00.143668    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="da9d25f0-e77a-46e2-b08f-b596242978d1" containerName="main"
Jul 30 00:45:00.143800 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:00.143730    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="da9d25f0-e77a-46e2-b08f-b596242978d1" containerName="main"
Jul 30 00:45:00.143800 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:00.143736    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ecb1f1f-c149-4269-b9d0-e642e8f01853" containerName="main"
Jul 30 00:45:00.146640 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:00.146612    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756205-pr8xl"
Jul 30 00:45:00.149666 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:00.149639    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v8q7j\""
Jul 30 00:45:00.156216 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:00.156200    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756205-pr8xl"
Jul 30 00:45:00.166751 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:00.166700    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756205-pr8xl"]
Jul 30 00:45:00.288185 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:00.288163    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756205-pr8xl"]
Jul 30 00:45:00.290290 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:45:00.290260    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa35fdf6_d72f_4132_8a4c_5486c5f819d2.slice/crio-6e1c6169339846d58ea2489ea5bc2ff1e75f0206b66e7afe406eea583209d2c8 WatchSource:0}: Error finding container 6e1c6169339846d58ea2489ea5bc2ff1e75f0206b66e7afe406eea583209d2c8: Status 404 returned error can't find the container with id 6e1c6169339846d58ea2489ea5bc2ff1e75f0206b66e7afe406eea583209d2c8
Jul 30 00:45:00.292298 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:00.292281    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 00:45:00.843540 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:00.843503    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756205-pr8xl" event={"ID":"fa35fdf6-d72f-4132-8a4c-5486c5f819d2","Type":"ContainerStarted","Data":"6e1c6169339846d58ea2489ea5bc2ff1e75f0206b66e7afe406eea583209d2c8"}
Jul 30 00:45:03.864360 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:03.864323    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756205-pr8xl" event={"ID":"fa35fdf6-d72f-4132-8a4c-5486c5f819d2","Type":"ContainerStarted","Data":"9acf14fb0771f6e6b9d99eb2a9fe6bacf393eecf4d7de4f093d76b0452abcb83"}
Jul 30 00:45:03.879155 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:03.879107    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756205-pr8xl" podStartSLOduration=1.097663456 podStartE2EDuration="3.879089782s" totalImagesPullingTime="2.781426326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:45:03.878902168 +0000 UTC m=+1629.176169102" watchObservedRunningTime="2026-07-30 00:45:03.879089782 +0000 UTC m=+1629.176356713"
Jul 30 00:45:04.870427 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:04.870391    2579 generic.go:356] "Generic (PLEG): container finished" podID="fa35fdf6-d72f-4132-8a4c-5486c5f819d2" containerID="9acf14fb0771f6e6b9d99eb2a9fe6bacf393eecf4d7de4f093d76b0452abcb83" exitCode=0
Jul 30 00:45:04.870831 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:04.870438    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756205-pr8xl" event={"ID":"fa35fdf6-d72f-4132-8a4c-5486c5f819d2","Type":"ContainerDied","Data":"9acf14fb0771f6e6b9d99eb2a9fe6bacf393eecf4d7de4f093d76b0452abcb83"}
Jul 30 00:45:06.004059 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:06.004035    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756205-pr8xl"
Jul 30 00:45:06.881475 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:06.881451    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756205-pr8xl"
Jul 30 00:45:06.881637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:06.881498    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756205-pr8xl" event={"ID":"fa35fdf6-d72f-4132-8a4c-5486c5f819d2","Type":"ContainerDied","Data":"6e1c6169339846d58ea2489ea5bc2ff1e75f0206b66e7afe406eea583209d2c8"}
Jul 30 00:45:06.881637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:06.881538    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e1c6169339846d58ea2489ea5bc2ff1e75f0206b66e7afe406eea583209d2c8"
Jul 30 00:45:39.588557 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:39.588522    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"]
Jul 30 00:45:39.589170 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:39.588921    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa35fdf6-d72f-4132-8a4c-5486c5f819d2" containerName="cleanup"
Jul 30 00:45:39.589170 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:39.588932    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa35fdf6-d72f-4132-8a4c-5486c5f819d2" containerName="cleanup"
Jul 30 00:45:39.589170 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:39.588999    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa35fdf6-d72f-4132-8a4c-5486c5f819d2" containerName="cleanup"
Jul 30 00:45:39.596465 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:39.596441    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"
Jul 30 00:45:39.599398 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:39.599374    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a5930c62-serving-cert\""
Jul 30 00:45:39.601851 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:39.601828    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"]
Jul 30 00:45:39.730876 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:39.730849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3256e525-a528-4b7f-b253-d729eace7a80-maas-api-tls\") pod \"maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l\" (UID: \"3256e525-a528-4b7f-b253-d729eace7a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"
Jul 30 00:45:39.731018 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:39.730889    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3256e525-a528-4b7f-b253-d729eace7a80-openshift-service-ca\") pod \"maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l\" (UID: \"3256e525-a528-4b7f-b253-d729eace7a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"
Jul 30 00:45:39.731062 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:39.731015    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhhj4\" (UniqueName: \"kubernetes.io/projected/3256e525-a528-4b7f-b253-d729eace7a80-kube-api-access-lhhj4\") pod \"maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l\" (UID: \"3256e525-a528-4b7f-b253-d729eace7a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"
Jul 30 00:45:39.831565 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:39.831537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhhj4\" (UniqueName: \"kubernetes.io/projected/3256e525-a528-4b7f-b253-d729eace7a80-kube-api-access-lhhj4\") pod \"maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l\" (UID: \"3256e525-a528-4b7f-b253-d729eace7a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"
Jul 30 00:45:39.831701 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:39.831575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3256e525-a528-4b7f-b253-d729eace7a80-maas-api-tls\") pod \"maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l\" (UID: \"3256e525-a528-4b7f-b253-d729eace7a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"
Jul 30 00:45:39.831701 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:39.831631    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3256e525-a528-4b7f-b253-d729eace7a80-openshift-service-ca\") pod \"maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l\" (UID: \"3256e525-a528-4b7f-b253-d729eace7a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"
Jul 30 00:45:39.832235 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:39.832203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3256e525-a528-4b7f-b253-d729eace7a80-openshift-service-ca\") pod \"maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l\" (UID: \"3256e525-a528-4b7f-b253-d729eace7a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"
Jul 30 00:45:39.833864 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:39.833845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3256e525-a528-4b7f-b253-d729eace7a80-maas-api-tls\") pod \"maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l\" (UID: \"3256e525-a528-4b7f-b253-d729eace7a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"
Jul 30 00:45:39.839188 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:39.839129    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhhj4\" (UniqueName: \"kubernetes.io/projected/3256e525-a528-4b7f-b253-d729eace7a80-kube-api-access-lhhj4\") pod \"maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l\" (UID: \"3256e525-a528-4b7f-b253-d729eace7a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"
Jul 30 00:45:39.909684 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:39.909656    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"
Jul 30 00:45:40.047395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.047311    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"]
Jul 30 00:45:40.049700 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:45:40.049669    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3256e525_a528_4b7f_b253_d729eace7a80.slice/crio-0e4c5c306e9867982046e5b9a21f664ce1c479745fd338b4a607df2e42e948f7 WatchSource:0}: Error finding container 0e4c5c306e9867982046e5b9a21f664ce1c479745fd338b4a607df2e42e948f7: Status 404 returned error can't find the container with id 0e4c5c306e9867982046e5b9a21f664ce1c479745fd338b4a607df2e42e948f7
Jul 30 00:45:40.142376 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.142348    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h"]
Jul 30 00:45:40.147377 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.147360    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h"
Jul 30 00:45:40.151701 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.151681    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a5930c62\""
Jul 30 00:45:40.151809 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.151726    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a5930c62-dockercfg-8krld\""
Jul 30 00:45:40.159552 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.159527    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h"]
Jul 30 00:45:40.177800 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.177770    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr"]
Jul 30 00:45:40.181309 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.181290    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr"
Jul 30 00:45:40.191067 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.191046    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr"]
Jul 30 00:45:40.235722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.235700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k757q\" (UniqueName: \"kubernetes.io/projected/9cada469-23b5-4dff-85e5-4ca92dd4b6b9-kube-api-access-k757q\") pod \"payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h\" (UID: \"9cada469-23b5-4dff-85e5-4ca92dd4b6b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h"
Jul 30 00:45:40.235845 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.235741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cada469-23b5-4dff-85e5-4ca92dd4b6b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h\" (UID: \"9cada469-23b5-4dff-85e5-4ca92dd4b6b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h"
Jul 30 00:45:40.336394 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.336367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2s2h\" (UniqueName: \"kubernetes.io/projected/76626af9-001a-4474-aafa-dc628efe92b5-kube-api-access-n2s2h\") pod \"payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr\" (UID: \"76626af9-001a-4474-aafa-dc628efe92b5\") " pod="openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr"
Jul 30 00:45:40.336572 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.336414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k757q\" (UniqueName: \"kubernetes.io/projected/9cada469-23b5-4dff-85e5-4ca92dd4b6b9-kube-api-access-k757q\") pod \"payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h\" (UID: \"9cada469-23b5-4dff-85e5-4ca92dd4b6b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h"
Jul 30 00:45:40.336572 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.336474    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76626af9-001a-4474-aafa-dc628efe92b5-plugins-config-volume\") pod \"payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr\" (UID: \"76626af9-001a-4474-aafa-dc628efe92b5\") " pod="openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr"
Jul 30 00:45:40.336572 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.336516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cada469-23b5-4dff-85e5-4ca92dd4b6b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h\" (UID: \"9cada469-23b5-4dff-85e5-4ca92dd4b6b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h"
Jul 30 00:45:40.337149 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.337132    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cada469-23b5-4dff-85e5-4ca92dd4b6b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h\" (UID: \"9cada469-23b5-4dff-85e5-4ca92dd4b6b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h"
Jul 30 00:45:40.345576 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.345551    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k757q\" (UniqueName: \"kubernetes.io/projected/9cada469-23b5-4dff-85e5-4ca92dd4b6b9-kube-api-access-k757q\") pod \"payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h\" (UID: \"9cada469-23b5-4dff-85e5-4ca92dd4b6b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h"
Jul 30 00:45:40.437171 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.437090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2s2h\" (UniqueName: \"kubernetes.io/projected/76626af9-001a-4474-aafa-dc628efe92b5-kube-api-access-n2s2h\") pod \"payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr\" (UID: \"76626af9-001a-4474-aafa-dc628efe92b5\") " pod="openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr"
Jul 30 00:45:40.437171 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.437137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76626af9-001a-4474-aafa-dc628efe92b5-plugins-config-volume\") pod \"payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr\" (UID: \"76626af9-001a-4474-aafa-dc628efe92b5\") " pod="openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr"
Jul 30 00:45:40.437873 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.437849    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76626af9-001a-4474-aafa-dc628efe92b5-plugins-config-volume\") pod \"payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr\" (UID: \"76626af9-001a-4474-aafa-dc628efe92b5\") " pod="openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr"
Jul 30 00:45:40.447496 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.447470    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2s2h\" (UniqueName: \"kubernetes.io/projected/76626af9-001a-4474-aafa-dc628efe92b5-kube-api-access-n2s2h\") pod \"payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr\" (UID: \"76626af9-001a-4474-aafa-dc628efe92b5\") " pod="openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr"
Jul 30 00:45:40.458305 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.458281    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h"
Jul 30 00:45:40.492030 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.491994    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr"
Jul 30 00:45:40.809667 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.809465    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h"]
Jul 30 00:45:40.812303 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:45:40.812277    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cada469_23b5_4dff_85e5_4ca92dd4b6b9.slice/crio-e6063080f5581d022d6cfbf61fd643158be2228717119692a8b6fd27440a1613 WatchSource:0}: Error finding container e6063080f5581d022d6cfbf61fd643158be2228717119692a8b6fd27440a1613: Status 404 returned error can't find the container with id e6063080f5581d022d6cfbf61fd643158be2228717119692a8b6fd27440a1613
Jul 30 00:45:40.846091 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:40.846067    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr"]
Jul 30 00:45:40.847755 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:45:40.847727    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76626af9_001a_4474_aafa_dc628efe92b5.slice/crio-242ff9ff7dfdb8e8444657a144b974f3314e35c4409bdaaf3b5194650ac71b73 WatchSource:0}: Error finding container 242ff9ff7dfdb8e8444657a144b974f3314e35c4409bdaaf3b5194650ac71b73: Status 404 returned error can't find the container with id 242ff9ff7dfdb8e8444657a144b974f3314e35c4409bdaaf3b5194650ac71b73
Jul 30 00:45:41.051150 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:41.051113    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l" event={"ID":"3256e525-a528-4b7f-b253-d729eace7a80","Type":"ContainerStarted","Data":"9d346b1d24ce52d65d02e54f3ce64de06892f819697fb6d10e0dbe0bdda68236"}
Jul 30 00:45:41.051366 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:41.051174    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l" event={"ID":"3256e525-a528-4b7f-b253-d729eace7a80","Type":"ContainerStarted","Data":"0e4c5c306e9867982046e5b9a21f664ce1c479745fd338b4a607df2e42e948f7"}
Jul 30 00:45:41.051366 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:41.051232    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"
Jul 30 00:45:41.053106 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:41.053074    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h" event={"ID":"9cada469-23b5-4dff-85e5-4ca92dd4b6b9","Type":"ContainerStarted","Data":"7dc0d52ac5b46d7e31cb0ca0a04b0707975b7c91908f2c2f2cb89e3c371f4bf1"}
Jul 30 00:45:41.053305 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:41.053130    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h" event={"ID":"9cada469-23b5-4dff-85e5-4ca92dd4b6b9","Type":"ContainerStarted","Data":"e6063080f5581d022d6cfbf61fd643158be2228717119692a8b6fd27440a1613"}
Jul 30 00:45:41.053305 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:41.053161    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h"
Jul 30 00:45:41.054949 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:41.054916    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr" event={"ID":"76626af9-001a-4474-aafa-dc628efe92b5","Type":"ContainerStarted","Data":"2f7f6aef5a5a6eda072f9d39517f1766a6867aca6ef7083cbfc2d323f8dd7d59"}
Jul 30 00:45:41.055069 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:41.054967    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr" event={"ID":"76626af9-001a-4474-aafa-dc628efe92b5","Type":"ContainerStarted","Data":"242ff9ff7dfdb8e8444657a144b974f3314e35c4409bdaaf3b5194650ac71b73"}
Jul 30 00:45:41.055069 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:41.054997    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr"
Jul 30 00:45:41.067157 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:41.067066    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l" podStartSLOduration=1.534551716 podStartE2EDuration="2.06704852s" totalImagesPullingTime="532.496804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:45:41.066754099 +0000 UTC m=+1666.364021033" watchObservedRunningTime="2026-07-30 00:45:41.06704852 +0000 UTC m=+1666.364315452"
Jul 30 00:45:41.082989 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:41.082950    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h" podStartSLOduration=1.082937676 podStartE2EDuration="1.082937676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:45:41.081232671 +0000 UTC m=+1666.378499603" watchObservedRunningTime="2026-07-30 00:45:41.082937676 +0000 UTC m=+1666.380204608"
Jul 30 00:45:41.097785 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:41.097740    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr" podStartSLOduration=1.097728473 podStartE2EDuration="1.097728473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:45:41.096159203 +0000 UTC m=+1666.393426136" watchObservedRunningTime="2026-07-30 00:45:41.097728473 +0000 UTC m=+1666.394995405"
Jul 30 00:45:47.066016 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:47.065981    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"
Jul 30 00:45:52.063170 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:52.063138    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr"
Jul 30 00:45:52.063584 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:45:52.063203    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h"
Jul 30 00:46:03.725982 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:03.725944    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5930c62-gmz9k"]
Jul 30 00:46:03.729496 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:03.729474    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5930c62-gmz9k"
Jul 30 00:46:03.732044 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:03.732023    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v8q7j\""
Jul 30 00:46:03.736526 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:03.736499    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5930c62-gmz9k"]
Jul 30 00:46:03.844542 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:03.844487    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c26900-588b-4cff-aac7-040c5bb72cd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5930c62-gmz9k\" (UID: \"61c26900-588b-4cff-aac7-040c5bb72cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5930c62-gmz9k"
Jul 30 00:46:03.945772 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:03.945743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c26900-588b-4cff-aac7-040c5bb72cd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5930c62-gmz9k\" (UID: \"61c26900-588b-4cff-aac7-040c5bb72cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5930c62-gmz9k"
Jul 30 00:46:03.946361 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:03.946344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c26900-588b-4cff-aac7-040c5bb72cd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5930c62-gmz9k\" (UID: \"61c26900-588b-4cff-aac7-040c5bb72cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5930c62-gmz9k"
Jul 30 00:46:04.041853 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:04.041789    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5930c62-gmz9k"
Jul 30 00:46:04.177940 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:04.177916    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5930c62-gmz9k"]
Jul 30 00:46:04.181370 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:46:04.181238    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c26900_588b_4cff_aac7_040c5bb72cd6.slice/crio-afe726434b9b235947ee3286d6f8d0ad385bae5f186a4cb439891ffb52eb44e8 WatchSource:0}: Error finding container afe726434b9b235947ee3286d6f8d0ad385bae5f186a4cb439891ffb52eb44e8: Status 404 returned error can't find the container with id afe726434b9b235947ee3286d6f8d0ad385bae5f186a4cb439891ffb52eb44e8
Jul 30 00:46:05.179361 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:05.179327    2579 generic.go:356] "Generic (PLEG): container finished" podID="61c26900-588b-4cff-aac7-040c5bb72cd6" containerID="acd5f4658c1a68a93397a8f276cc419c46b26cb4ce2ae105675d9fb576e4a152" exitCode=0
Jul 30 00:46:05.179786 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:05.179413    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5930c62-gmz9k" event={"ID":"61c26900-588b-4cff-aac7-040c5bb72cd6","Type":"ContainerDied","Data":"acd5f4658c1a68a93397a8f276cc419c46b26cb4ce2ae105675d9fb576e4a152"}
Jul 30 00:46:05.179786 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:05.179468    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5930c62-gmz9k" event={"ID":"61c26900-588b-4cff-aac7-040c5bb72cd6","Type":"ContainerStarted","Data":"afe726434b9b235947ee3286d6f8d0ad385bae5f186a4cb439891ffb52eb44e8"}
Jul 30 00:46:06.311617 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:06.311595    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5930c62-gmz9k"
Jul 30 00:46:06.369945 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:06.369923    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c26900-588b-4cff-aac7-040c5bb72cd6-maas-api-service-ca\") pod \"61c26900-588b-4cff-aac7-040c5bb72cd6\" (UID: \"61c26900-588b-4cff-aac7-040c5bb72cd6\") "
Jul 30 00:46:06.370254 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:06.370221    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c26900-588b-4cff-aac7-040c5bb72cd6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "61c26900-588b-4cff-aac7-040c5bb72cd6" (UID: "61c26900-588b-4cff-aac7-040c5bb72cd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:46:06.471202 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:06.471129    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c26900-588b-4cff-aac7-040c5bb72cd6-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:46:07.190048 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:07.190012    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5930c62-gmz9k" event={"ID":"61c26900-588b-4cff-aac7-040c5bb72cd6","Type":"ContainerDied","Data":"afe726434b9b235947ee3286d6f8d0ad385bae5f186a4cb439891ffb52eb44e8"}
Jul 30 00:46:07.190048 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:07.190036    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5930c62-gmz9k"
Jul 30 00:46:07.190283 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:07.190069    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afe726434b9b235947ee3286d6f8d0ad385bae5f186a4cb439891ffb52eb44e8"
Jul 30 00:46:09.985503 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:09.985469    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"]
Jul 30 00:46:09.986119 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:09.985681    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l" podUID="3256e525-a528-4b7f-b253-d729eace7a80" containerName="maas-api" containerID="cri-o://9d346b1d24ce52d65d02e54f3ce64de06892f819697fb6d10e0dbe0bdda68236" gracePeriod=30
Jul 30 00:46:10.090614 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.090581    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr"]
Jul 30 00:46:10.090840 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.090809    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr" podUID="76626af9-001a-4474-aafa-dc628efe92b5" containerName="payload-processing" containerID="cri-o://2f7f6aef5a5a6eda072f9d39517f1766a6867aca6ef7083cbfc2d323f8dd7d59" gracePeriod=30
Jul 30 00:46:10.098024 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.097996    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h"]
Jul 30 00:46:10.098351 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.098319    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h" podUID="9cada469-23b5-4dff-85e5-4ca92dd4b6b9" containerName="payload-pre-processing" containerID="cri-o://7dc0d52ac5b46d7e31cb0ca0a04b0707975b7c91908f2c2f2cb89e3c371f4bf1" gracePeriod=30
Jul 30 00:46:10.210326 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.210291    2579 generic.go:356] "Generic (PLEG): container finished" podID="3256e525-a528-4b7f-b253-d729eace7a80" containerID="9d346b1d24ce52d65d02e54f3ce64de06892f819697fb6d10e0dbe0bdda68236" exitCode=0
Jul 30 00:46:10.210453 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.210354    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l" event={"ID":"3256e525-a528-4b7f-b253-d729eace7a80","Type":"ContainerDied","Data":"9d346b1d24ce52d65d02e54f3ce64de06892f819697fb6d10e0dbe0bdda68236"}
Jul 30 00:46:10.212443 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.212413    2579 generic.go:356] "Generic (PLEG): container finished" podID="76626af9-001a-4474-aafa-dc628efe92b5" containerID="2f7f6aef5a5a6eda072f9d39517f1766a6867aca6ef7083cbfc2d323f8dd7d59" exitCode=0
Jul 30 00:46:10.212443 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.212431    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr" event={"ID":"76626af9-001a-4474-aafa-dc628efe92b5","Type":"ContainerDied","Data":"2f7f6aef5a5a6eda072f9d39517f1766a6867aca6ef7083cbfc2d323f8dd7d59"}
Jul 30 00:46:10.239887 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.238906    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5930c62-gmz9k"]
Jul 30 00:46:10.245554 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.245491    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5930c62-gmz9k"]
Jul 30 00:46:10.260282 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.260219    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"
Jul 30 00:46:10.305658 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.305635    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3256e525-a528-4b7f-b253-d729eace7a80-maas-api-tls\") pod \"3256e525-a528-4b7f-b253-d729eace7a80\" (UID: \"3256e525-a528-4b7f-b253-d729eace7a80\") "
Jul 30 00:46:10.305755 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.305694    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhhj4\" (UniqueName: \"kubernetes.io/projected/3256e525-a528-4b7f-b253-d729eace7a80-kube-api-access-lhhj4\") pod \"3256e525-a528-4b7f-b253-d729eace7a80\" (UID: \"3256e525-a528-4b7f-b253-d729eace7a80\") "
Jul 30 00:46:10.305823 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.305756    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3256e525-a528-4b7f-b253-d729eace7a80-openshift-service-ca\") pod \"3256e525-a528-4b7f-b253-d729eace7a80\" (UID: \"3256e525-a528-4b7f-b253-d729eace7a80\") "
Jul 30 00:46:10.306122 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.306097    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3256e525-a528-4b7f-b253-d729eace7a80-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3256e525-a528-4b7f-b253-d729eace7a80" (UID: "3256e525-a528-4b7f-b253-d729eace7a80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:46:10.308055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.308029    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3256e525-a528-4b7f-b253-d729eace7a80-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3256e525-a528-4b7f-b253-d729eace7a80" (UID: "3256e525-a528-4b7f-b253-d729eace7a80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:46:10.308189 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.308164    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3256e525-a528-4b7f-b253-d729eace7a80-kube-api-access-lhhj4" (OuterVolumeSpecName: "kube-api-access-lhhj4") pod "3256e525-a528-4b7f-b253-d729eace7a80" (UID: "3256e525-a528-4b7f-b253-d729eace7a80"). InnerVolumeSpecName "kube-api-access-lhhj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:46:10.369144 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.369119    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr"
Jul 30 00:46:10.383811 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.383787    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h"
Jul 30 00:46:10.406761 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.406737    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76626af9-001a-4474-aafa-dc628efe92b5-plugins-config-volume\") pod \"76626af9-001a-4474-aafa-dc628efe92b5\" (UID: \"76626af9-001a-4474-aafa-dc628efe92b5\") "
Jul 30 00:46:10.406912 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.406770    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2s2h\" (UniqueName: \"kubernetes.io/projected/76626af9-001a-4474-aafa-dc628efe92b5-kube-api-access-n2s2h\") pod \"76626af9-001a-4474-aafa-dc628efe92b5\" (UID: \"76626af9-001a-4474-aafa-dc628efe92b5\") "
Jul 30 00:46:10.407047 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.407030    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3256e525-a528-4b7f-b253-d729eace7a80-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:46:10.407091 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.407056    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhhj4\" (UniqueName: \"kubernetes.io/projected/3256e525-a528-4b7f-b253-d729eace7a80-kube-api-access-lhhj4\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:46:10.407091 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.407072    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3256e525-a528-4b7f-b253-d729eace7a80-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:46:10.407091 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.407078    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76626af9-001a-4474-aafa-dc628efe92b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "76626af9-001a-4474-aafa-dc628efe92b5" (UID: "76626af9-001a-4474-aafa-dc628efe92b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:46:10.409120 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.409097    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76626af9-001a-4474-aafa-dc628efe92b5-kube-api-access-n2s2h" (OuterVolumeSpecName: "kube-api-access-n2s2h") pod "76626af9-001a-4474-aafa-dc628efe92b5" (UID: "76626af9-001a-4474-aafa-dc628efe92b5"). InnerVolumeSpecName "kube-api-access-n2s2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:46:10.507835 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.507768    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k757q\" (UniqueName: \"kubernetes.io/projected/9cada469-23b5-4dff-85e5-4ca92dd4b6b9-kube-api-access-k757q\") pod \"9cada469-23b5-4dff-85e5-4ca92dd4b6b9\" (UID: \"9cada469-23b5-4dff-85e5-4ca92dd4b6b9\") "
Jul 30 00:46:10.507957 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.507863    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cada469-23b5-4dff-85e5-4ca92dd4b6b9-plugins-config-volume\") pod \"9cada469-23b5-4dff-85e5-4ca92dd4b6b9\" (UID: \"9cada469-23b5-4dff-85e5-4ca92dd4b6b9\") "
Jul 30 00:46:10.508022 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.508011    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76626af9-001a-4474-aafa-dc628efe92b5-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:46:10.508061 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.508045    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2s2h\" (UniqueName: \"kubernetes.io/projected/76626af9-001a-4474-aafa-dc628efe92b5-kube-api-access-n2s2h\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:46:10.508197 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.508176    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cada469-23b5-4dff-85e5-4ca92dd4b6b9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9cada469-23b5-4dff-85e5-4ca92dd4b6b9" (UID: "9cada469-23b5-4dff-85e5-4ca92dd4b6b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:46:10.509750 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.509731    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cada469-23b5-4dff-85e5-4ca92dd4b6b9-kube-api-access-k757q" (OuterVolumeSpecName: "kube-api-access-k757q") pod "9cada469-23b5-4dff-85e5-4ca92dd4b6b9" (UID: "9cada469-23b5-4dff-85e5-4ca92dd4b6b9"). InnerVolumeSpecName "kube-api-access-k757q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:46:10.608957 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.608922    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cada469-23b5-4dff-85e5-4ca92dd4b6b9-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:46:10.609136 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:10.608963    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k757q\" (UniqueName: \"kubernetes.io/projected/9cada469-23b5-4dff-85e5-4ca92dd4b6b9-kube-api-access-k757q\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:46:11.218892 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.218860    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"
Jul 30 00:46:11.218892 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.218865    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l" event={"ID":"3256e525-a528-4b7f-b253-d729eace7a80","Type":"ContainerDied","Data":"0e4c5c306e9867982046e5b9a21f664ce1c479745fd338b4a607df2e42e948f7"}
Jul 30 00:46:11.219395 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.218926    2579 scope.go:117] "RemoveContainer" containerID="9d346b1d24ce52d65d02e54f3ce64de06892f819697fb6d10e0dbe0bdda68236"
Jul 30 00:46:11.220283 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.220261    2579 generic.go:356] "Generic (PLEG): container finished" podID="9cada469-23b5-4dff-85e5-4ca92dd4b6b9" containerID="7dc0d52ac5b46d7e31cb0ca0a04b0707975b7c91908f2c2f2cb89e3c371f4bf1" exitCode=0
Jul 30 00:46:11.220370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.220302    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h"
Jul 30 00:46:11.220370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.220335    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h" event={"ID":"9cada469-23b5-4dff-85e5-4ca92dd4b6b9","Type":"ContainerDied","Data":"7dc0d52ac5b46d7e31cb0ca0a04b0707975b7c91908f2c2f2cb89e3c371f4bf1"}
Jul 30 00:46:11.220480 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.220389    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h" event={"ID":"9cada469-23b5-4dff-85e5-4ca92dd4b6b9","Type":"ContainerDied","Data":"e6063080f5581d022d6cfbf61fd643158be2228717119692a8b6fd27440a1613"}
Jul 30 00:46:11.222025 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.222005    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr"
Jul 30 00:46:11.222151 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.222037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr" event={"ID":"76626af9-001a-4474-aafa-dc628efe92b5","Type":"ContainerDied","Data":"242ff9ff7dfdb8e8444657a144b974f3314e35c4409bdaaf3b5194650ac71b73"}
Jul 30 00:46:11.230117 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.230097    2579 scope.go:117] "RemoveContainer" containerID="7dc0d52ac5b46d7e31cb0ca0a04b0707975b7c91908f2c2f2cb89e3c371f4bf1"
Jul 30 00:46:11.238785 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.238770    2579 scope.go:117] "RemoveContainer" containerID="7dc0d52ac5b46d7e31cb0ca0a04b0707975b7c91908f2c2f2cb89e3c371f4bf1"
Jul 30 00:46:11.239075 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:46:11.239036    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc0d52ac5b46d7e31cb0ca0a04b0707975b7c91908f2c2f2cb89e3c371f4bf1\": container with ID starting with 7dc0d52ac5b46d7e31cb0ca0a04b0707975b7c91908f2c2f2cb89e3c371f4bf1 not found: ID does not exist" containerID="7dc0d52ac5b46d7e31cb0ca0a04b0707975b7c91908f2c2f2cb89e3c371f4bf1"
Jul 30 00:46:11.239161 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.239077    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc0d52ac5b46d7e31cb0ca0a04b0707975b7c91908f2c2f2cb89e3c371f4bf1"} err="failed to get container status \"7dc0d52ac5b46d7e31cb0ca0a04b0707975b7c91908f2c2f2cb89e3c371f4bf1\": rpc error: code = NotFound desc = could not find container \"7dc0d52ac5b46d7e31cb0ca0a04b0707975b7c91908f2c2f2cb89e3c371f4bf1\": container with ID starting with 7dc0d52ac5b46d7e31cb0ca0a04b0707975b7c91908f2c2f2cb89e3c371f4bf1 not found: ID does not exist"
Jul 30 00:46:11.239161 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.239102    2579 scope.go:117] "RemoveContainer" containerID="2f7f6aef5a5a6eda072f9d39517f1766a6867aca6ef7083cbfc2d323f8dd7d59"
Jul 30 00:46:11.250782 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.250727    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"]
Jul 30 00:46:11.257458 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.257436    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a5930c62-7b9976d578-fnv8l"]
Jul 30 00:46:11.271677 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.271657    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h"]
Jul 30 00:46:11.280932 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.280912    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a5930c62-d8c6f4cf6-9mv2h"]
Jul 30 00:46:11.295875 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.295851    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr"]
Jul 30 00:46:11.303290 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.303267    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a5930c62-745bcd66b-h84fr"]
Jul 30 00:46:11.323399 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.323373    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3256e525-a528-4b7f-b253-d729eace7a80" path="/var/lib/kubelet/pods/3256e525-a528-4b7f-b253-d729eace7a80/volumes"
Jul 30 00:46:11.323710 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.323691    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61c26900-588b-4cff-aac7-040c5bb72cd6" path="/var/lib/kubelet/pods/61c26900-588b-4cff-aac7-040c5bb72cd6/volumes"
Jul 30 00:46:11.323997 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.323985    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76626af9-001a-4474-aafa-dc628efe92b5" path="/var/lib/kubelet/pods/76626af9-001a-4474-aafa-dc628efe92b5/volumes"
Jul 30 00:46:11.324356 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:11.324341    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cada469-23b5-4dff-85e5-4ca92dd4b6b9" path="/var/lib/kubelet/pods/9cada469-23b5-4dff-85e5-4ca92dd4b6b9/volumes"
Jul 30 00:46:13.675564 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.675528    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"]
Jul 30 00:46:13.675967 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.675952    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3256e525-a528-4b7f-b253-d729eace7a80" containerName="maas-api"
Jul 30 00:46:13.676023 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.675970    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3256e525-a528-4b7f-b253-d729eace7a80" containerName="maas-api"
Jul 30 00:46:13.676023 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.675982    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cada469-23b5-4dff-85e5-4ca92dd4b6b9" containerName="payload-pre-processing"
Jul 30 00:46:13.676023 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.675988    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cada469-23b5-4dff-85e5-4ca92dd4b6b9" containerName="payload-pre-processing"
Jul 30 00:46:13.676023 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.675995    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61c26900-588b-4cff-aac7-040c5bb72cd6" containerName="revoke-keys"
Jul 30 00:46:13.676023 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.676002    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="61c26900-588b-4cff-aac7-040c5bb72cd6" containerName="revoke-keys"
Jul 30 00:46:13.676179 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.676025    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76626af9-001a-4474-aafa-dc628efe92b5" containerName="payload-processing"
Jul 30 00:46:13.676179 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.676031    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="76626af9-001a-4474-aafa-dc628efe92b5" containerName="payload-processing"
Jul 30 00:46:13.676179 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.676101    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="61c26900-588b-4cff-aac7-040c5bb72cd6" containerName="revoke-keys"
Jul 30 00:46:13.676179 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.676110    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3256e525-a528-4b7f-b253-d729eace7a80" containerName="maas-api"
Jul 30 00:46:13.676179 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.676116    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cada469-23b5-4dff-85e5-4ca92dd4b6b9" containerName="payload-pre-processing"
Jul 30 00:46:13.676179 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.676122    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="76626af9-001a-4474-aafa-dc628efe92b5" containerName="payload-processing"
Jul 30 00:46:13.680748 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.680729    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"
Jul 30 00:46:13.683300 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.683280    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-837664ed-serving-cert\""
Jul 30 00:46:13.687068 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.687010    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"]
Jul 30 00:46:13.736627 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.736602    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-openshift-service-ca\") pod \"maas-api-e2e-migrate-837664ed-db878468c-jtbhx\" (UID: \"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"
Jul 30 00:46:13.736769 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.736635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w5dm\" (UniqueName: \"kubernetes.io/projected/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-kube-api-access-6w5dm\") pod \"maas-api-e2e-migrate-837664ed-db878468c-jtbhx\" (UID: \"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"
Jul 30 00:46:13.736769 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.736654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-maas-api-tls\") pod \"maas-api-e2e-migrate-837664ed-db878468c-jtbhx\" (UID: \"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"
Jul 30 00:46:13.838094 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.838069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-openshift-service-ca\") pod \"maas-api-e2e-migrate-837664ed-db878468c-jtbhx\" (UID: \"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"
Jul 30 00:46:13.838198 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.838104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w5dm\" (UniqueName: \"kubernetes.io/projected/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-kube-api-access-6w5dm\") pod \"maas-api-e2e-migrate-837664ed-db878468c-jtbhx\" (UID: \"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"
Jul 30 00:46:13.838198 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.838127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-maas-api-tls\") pod \"maas-api-e2e-migrate-837664ed-db878468c-jtbhx\" (UID: \"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"
Jul 30 00:46:13.838331 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:46:13.838300    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-serving-cert: secret "maas-api-e2e-migrate-837664ed-serving-cert" not found
Jul 30 00:46:13.838388 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:46:13.838376    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-maas-api-tls podName:f5b3d9b8-31b5-4fb0-aa82-8a888d69522d nodeName:}" failed. No retries permitted until 2026-07-30 00:46:14.338356785 +0000 UTC m=+1699.635623695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-maas-api-tls") pod "maas-api-e2e-migrate-837664ed-db878468c-jtbhx" (UID: "f5b3d9b8-31b5-4fb0-aa82-8a888d69522d") : secret "maas-api-e2e-migrate-837664ed-serving-cert" not found
Jul 30 00:46:13.838800 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.838785    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-openshift-service-ca\") pod \"maas-api-e2e-migrate-837664ed-db878468c-jtbhx\" (UID: \"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"
Jul 30 00:46:13.847066 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:13.847035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w5dm\" (UniqueName: \"kubernetes.io/projected/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-kube-api-access-6w5dm\") pod \"maas-api-e2e-migrate-837664ed-db878468c-jtbhx\" (UID: \"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"
Jul 30 00:46:14.292860 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.292828    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj"]
Jul 30 00:46:14.296430 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.296412    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj"
Jul 30 00:46:14.300795 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.300775    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-837664ed\""
Jul 30 00:46:14.301135 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.301113    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-837664ed-dockercfg-slw27\""
Jul 30 00:46:14.312543 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.312518    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj"]
Jul 30 00:46:14.330760 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.330739    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76"]
Jul 30 00:46:14.334030 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.334015    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76"
Jul 30 00:46:14.343586 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.343560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-maas-api-tls\") pod \"maas-api-e2e-migrate-837664ed-db878468c-jtbhx\" (UID: \"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"
Jul 30 00:46:14.343717 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.343609    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh9zf\" (UniqueName: \"kubernetes.io/projected/c73153c5-315e-4d14-9d82-f39b524c3906-kube-api-access-hh9zf\") pod \"payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj\" (UID: \"c73153c5-315e-4d14-9d82-f39b524c3906\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj"
Jul 30 00:46:14.343790 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.343763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73153c5-315e-4d14-9d82-f39b524c3906-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj\" (UID: \"c73153c5-315e-4d14-9d82-f39b524c3906\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj"
Jul 30 00:46:14.344062 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.344038    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76"]
Jul 30 00:46:14.346505 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.346483    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-maas-api-tls\") pod \"maas-api-e2e-migrate-837664ed-db878468c-jtbhx\" (UID: \"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"
Jul 30 00:46:14.445081 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.445049    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bcd80d-e456-46bf-b99b-c17304171135-plugins-config-volume\") pod \"payload-processing-e2e-migrate-837664ed-5668ffdf-prz76\" (UID: \"02bcd80d-e456-46bf-b99b-c17304171135\") " pod="openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76"
Jul 30 00:46:14.445289 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.445088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73153c5-315e-4d14-9d82-f39b524c3906-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj\" (UID: \"c73153c5-315e-4d14-9d82-f39b524c3906\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj"
Jul 30 00:46:14.445289 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.445109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwnsv\" (UniqueName: \"kubernetes.io/projected/02bcd80d-e456-46bf-b99b-c17304171135-kube-api-access-xwnsv\") pod \"payload-processing-e2e-migrate-837664ed-5668ffdf-prz76\" (UID: \"02bcd80d-e456-46bf-b99b-c17304171135\") " pod="openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76"
Jul 30 00:46:14.445289 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.445131    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9zf\" (UniqueName: \"kubernetes.io/projected/c73153c5-315e-4d14-9d82-f39b524c3906-kube-api-access-hh9zf\") pod \"payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj\" (UID: \"c73153c5-315e-4d14-9d82-f39b524c3906\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj"
Jul 30 00:46:14.445843 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.445816    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73153c5-315e-4d14-9d82-f39b524c3906-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj\" (UID: \"c73153c5-315e-4d14-9d82-f39b524c3906\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj"
Jul 30 00:46:14.453188 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.453166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh9zf\" (UniqueName: \"kubernetes.io/projected/c73153c5-315e-4d14-9d82-f39b524c3906-kube-api-access-hh9zf\") pod \"payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj\" (UID: \"c73153c5-315e-4d14-9d82-f39b524c3906\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj"
Jul 30 00:46:14.545818 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.545749    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwnsv\" (UniqueName: \"kubernetes.io/projected/02bcd80d-e456-46bf-b99b-c17304171135-kube-api-access-xwnsv\") pod \"payload-processing-e2e-migrate-837664ed-5668ffdf-prz76\" (UID: \"02bcd80d-e456-46bf-b99b-c17304171135\") " pod="openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76"
Jul 30 00:46:14.545940 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.545884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bcd80d-e456-46bf-b99b-c17304171135-plugins-config-volume\") pod \"payload-processing-e2e-migrate-837664ed-5668ffdf-prz76\" (UID: \"02bcd80d-e456-46bf-b99b-c17304171135\") " pod="openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76"
Jul 30 00:46:14.546585 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.546567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bcd80d-e456-46bf-b99b-c17304171135-plugins-config-volume\") pod \"payload-processing-e2e-migrate-837664ed-5668ffdf-prz76\" (UID: \"02bcd80d-e456-46bf-b99b-c17304171135\") " pod="openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76"
Jul 30 00:46:14.553727 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.553705    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwnsv\" (UniqueName: \"kubernetes.io/projected/02bcd80d-e456-46bf-b99b-c17304171135-kube-api-access-xwnsv\") pod \"payload-processing-e2e-migrate-837664ed-5668ffdf-prz76\" (UID: \"02bcd80d-e456-46bf-b99b-c17304171135\") " pod="openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76"
Jul 30 00:46:14.593704 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.593675    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"
Jul 30 00:46:14.607006 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.606978    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj"
Jul 30 00:46:14.646267 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.646222    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76"
Jul 30 00:46:14.748396 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.748200    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"]
Jul 30 00:46:14.754913 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:46:14.754865    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b3d9b8_31b5_4fb0_aa82_8a888d69522d.slice/crio-8b71c9e929fbf93de6e509feca5c5be0343b080798e9055a3c56d35806fd7859 WatchSource:0}: Error finding container 8b71c9e929fbf93de6e509feca5c5be0343b080798e9055a3c56d35806fd7859: Status 404 returned error can't find the container with id 8b71c9e929fbf93de6e509feca5c5be0343b080798e9055a3c56d35806fd7859
Jul 30 00:46:14.786849 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.786752    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj"]
Jul 30 00:46:14.789622 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:46:14.789579    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73153c5_315e_4d14_9d82_f39b524c3906.slice/crio-56acc3b7c1e4d725d68e720f79e3c435e86027ddbee31ed081bf061d629ea83b WatchSource:0}: Error finding container 56acc3b7c1e4d725d68e720f79e3c435e86027ddbee31ed081bf061d629ea83b: Status 404 returned error can't find the container with id 56acc3b7c1e4d725d68e720f79e3c435e86027ddbee31ed081bf061d629ea83b
Jul 30 00:46:14.818237 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:14.818204    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76"]
Jul 30 00:46:14.825393 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:46:14.825368    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bcd80d_e456_46bf_b99b_c17304171135.slice/crio-baa2f2bddceba743a738182a8d0fa7002989113d9750889e9f06df53637511dc WatchSource:0}: Error finding container baa2f2bddceba743a738182a8d0fa7002989113d9750889e9f06df53637511dc: Status 404 returned error can't find the container with id baa2f2bddceba743a738182a8d0fa7002989113d9750889e9f06df53637511dc
Jul 30 00:46:15.247454 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:15.247408    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj" event={"ID":"c73153c5-315e-4d14-9d82-f39b524c3906","Type":"ContainerStarted","Data":"7a01f6a52e15f2394811ac161d3a6eec6ac7b23c50eec4f2de9ec5841f683817"}
Jul 30 00:46:15.247659 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:15.247473    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj" event={"ID":"c73153c5-315e-4d14-9d82-f39b524c3906","Type":"ContainerStarted","Data":"56acc3b7c1e4d725d68e720f79e3c435e86027ddbee31ed081bf061d629ea83b"}
Jul 30 00:46:15.247659 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:15.247535    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj"
Jul 30 00:46:15.248741 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:15.248714    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx" event={"ID":"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d","Type":"ContainerStarted","Data":"8b71c9e929fbf93de6e509feca5c5be0343b080798e9055a3c56d35806fd7859"}
Jul 30 00:46:15.250424 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:15.250403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76" event={"ID":"02bcd80d-e456-46bf-b99b-c17304171135","Type":"ContainerStarted","Data":"f4d959b3952d543a133197f95377eb267e50dbf87032f3714c0e72d52c638b03"}
Jul 30 00:46:15.250546 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:15.250447    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76" event={"ID":"02bcd80d-e456-46bf-b99b-c17304171135","Type":"ContainerStarted","Data":"baa2f2bddceba743a738182a8d0fa7002989113d9750889e9f06df53637511dc"}
Jul 30 00:46:15.250546 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:15.250497    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76"
Jul 30 00:46:15.262946 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:15.262902    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj" podStartSLOduration=1.262884426 podStartE2EDuration="1.262884426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:46:15.261965309 +0000 UTC m=+1700.559232233" watchObservedRunningTime="2026-07-30 00:46:15.262884426 +0000 UTC m=+1700.560151357"
Jul 30 00:46:15.277817 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:15.277778    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76" podStartSLOduration=1.277764992 podStartE2EDuration="1.277764992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:46:15.277090714 +0000 UTC m=+1700.574357646" watchObservedRunningTime="2026-07-30 00:46:15.277764992 +0000 UTC m=+1700.575031924"
Jul 30 00:46:16.257884 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:16.257791    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx" event={"ID":"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d","Type":"ContainerStarted","Data":"641745ed17e130689c4045577234e73263272441cc5c45bf1c9c77ab8eb80d8e"}
Jul 30 00:46:16.276013 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:16.275960    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx" podStartSLOduration=2.178676888 podStartE2EDuration="3.275943897s" totalImagesPullingTime="1.097267009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:46:16.274034665 +0000 UTC m=+1701.571301598" watchObservedRunningTime="2026-07-30 00:46:16.275943897 +0000 UTC m=+1701.573210829"
Jul 30 00:46:17.264185 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:17.264152    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"
Jul 30 00:46:23.274770 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:23.274739    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"
Jul 30 00:46:26.259645 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:26.259608    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76"
Jul 30 00:46:26.261883 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:26.259684    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj"
Jul 30 00:46:38.558605 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:38.558569    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837664ed-9sh78"]
Jul 30 00:46:38.562275 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:38.562255    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837664ed-9sh78"
Jul 30 00:46:38.565121 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:38.565099    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v8q7j\""
Jul 30 00:46:38.570104 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:38.570080    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837664ed-9sh78"]
Jul 30 00:46:38.753439 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:38.753399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46ef4c5-739c-4bb3-afed-a976fa9983e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-837664ed-9sh78\" (UID: \"b46ef4c5-739c-4bb3-afed-a976fa9983e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837664ed-9sh78"
Jul 30 00:46:38.853979 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:38.853886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46ef4c5-739c-4bb3-afed-a976fa9983e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-837664ed-9sh78\" (UID: \"b46ef4c5-739c-4bb3-afed-a976fa9983e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837664ed-9sh78"
Jul 30 00:46:38.854542 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:38.854521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46ef4c5-739c-4bb3-afed-a976fa9983e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-837664ed-9sh78\" (UID: \"b46ef4c5-739c-4bb3-afed-a976fa9983e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837664ed-9sh78"
Jul 30 00:46:38.874877 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:38.874851    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837664ed-9sh78"
Jul 30 00:46:39.007720 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:39.007694    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837664ed-9sh78"]
Jul 30 00:46:39.009889 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:46:39.009844    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46ef4c5_739c_4bb3_afed_a976fa9983e8.slice/crio-541245fa5024c97916294d57b64f83f39022c501473b7bb94af7a38cc0a94e14 WatchSource:0}: Error finding container 541245fa5024c97916294d57b64f83f39022c501473b7bb94af7a38cc0a94e14: Status 404 returned error can't find the container with id 541245fa5024c97916294d57b64f83f39022c501473b7bb94af7a38cc0a94e14
Jul 30 00:46:39.376362 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:39.376332    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837664ed-9sh78" event={"ID":"b46ef4c5-739c-4bb3-afed-a976fa9983e8","Type":"ContainerStarted","Data":"fc3612f855e66a296f589d2b864cf4549b7bfede515427678cfa65a612af9d8f"}
Jul 30 00:46:39.376501 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:39.376378    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837664ed-9sh78" event={"ID":"b46ef4c5-739c-4bb3-afed-a976fa9983e8","Type":"ContainerStarted","Data":"541245fa5024c97916294d57b64f83f39022c501473b7bb94af7a38cc0a94e14"}
Jul 30 00:46:39.391226 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:39.391182    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837664ed-9sh78" podStartSLOduration=1.391165351 podStartE2EDuration="1.391165351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:46:39.39020042 +0000 UTC m=+1724.687467352" watchObservedRunningTime="2026-07-30 00:46:39.391165351 +0000 UTC m=+1724.688432282"
Jul 30 00:46:40.383187 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:40.383154    2579 generic.go:356] "Generic (PLEG): container finished" podID="b46ef4c5-739c-4bb3-afed-a976fa9983e8" containerID="fc3612f855e66a296f589d2b864cf4549b7bfede515427678cfa65a612af9d8f" exitCode=0
Jul 30 00:46:40.383602 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:40.383205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837664ed-9sh78" event={"ID":"b46ef4c5-739c-4bb3-afed-a976fa9983e8","Type":"ContainerDied","Data":"fc3612f855e66a296f589d2b864cf4549b7bfede515427678cfa65a612af9d8f"}
Jul 30 00:46:41.521879 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:41.521852    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837664ed-9sh78"
Jul 30 00:46:41.677562 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:41.677478    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46ef4c5-739c-4bb3-afed-a976fa9983e8-maas-api-service-ca\") pod \"b46ef4c5-739c-4bb3-afed-a976fa9983e8\" (UID: \"b46ef4c5-739c-4bb3-afed-a976fa9983e8\") "
Jul 30 00:46:41.677819 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:41.677790    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46ef4c5-739c-4bb3-afed-a976fa9983e8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b46ef4c5-739c-4bb3-afed-a976fa9983e8" (UID: "b46ef4c5-739c-4bb3-afed-a976fa9983e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:46:41.778307 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:41.778283    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46ef4c5-739c-4bb3-afed-a976fa9983e8-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:46:42.394542 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:42.394512    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837664ed-9sh78"
Jul 30 00:46:42.394705 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:42.394514    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837664ed-9sh78" event={"ID":"b46ef4c5-739c-4bb3-afed-a976fa9983e8","Type":"ContainerDied","Data":"541245fa5024c97916294d57b64f83f39022c501473b7bb94af7a38cc0a94e14"}
Jul 30 00:46:42.394705 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:42.394640    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="541245fa5024c97916294d57b64f83f39022c501473b7bb94af7a38cc0a94e14"
Jul 30 00:46:44.960762 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:44.960729    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"]
Jul 30 00:46:44.961275 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:44.960979    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx" podUID="f5b3d9b8-31b5-4fb0-aa82-8a888d69522d" containerName="maas-api" containerID="cri-o://641745ed17e130689c4045577234e73263272441cc5c45bf1c9c77ab8eb80d8e" gracePeriod=30
Jul 30 00:46:45.015335 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.015303    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76"]
Jul 30 00:46:45.015575 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.015548    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76" podUID="02bcd80d-e456-46bf-b99b-c17304171135" containerName="payload-processing" containerID="cri-o://f4d959b3952d543a133197f95377eb267e50dbf87032f3714c0e72d52c638b03" gracePeriod=30
Jul 30 00:46:45.022783 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.022756    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj"]
Jul 30 00:46:45.023006 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.022979    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj" podUID="c73153c5-315e-4d14-9d82-f39b524c3906" containerName="payload-pre-processing" containerID="cri-o://7a01f6a52e15f2394811ac161d3a6eec6ac7b23c50eec4f2de9ec5841f683817" gracePeriod=30
Jul 30 00:46:45.147465 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.147435    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837664ed-9sh78"]
Jul 30 00:46:45.152234 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.152204    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837664ed-9sh78"]
Jul 30 00:46:45.287768 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.287738    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"
Jul 30 00:46:45.291503 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.291483    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76"
Jul 30 00:46:45.308409 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.308324    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-openshift-service-ca\") pod \"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d\" (UID: \"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d\") "
Jul 30 00:46:45.308409 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.308386    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bcd80d-e456-46bf-b99b-c17304171135-plugins-config-volume\") pod \"02bcd80d-e456-46bf-b99b-c17304171135\" (UID: \"02bcd80d-e456-46bf-b99b-c17304171135\") "
Jul 30 00:46:45.308630 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.308459    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwnsv\" (UniqueName: \"kubernetes.io/projected/02bcd80d-e456-46bf-b99b-c17304171135-kube-api-access-xwnsv\") pod \"02bcd80d-e456-46bf-b99b-c17304171135\" (UID: \"02bcd80d-e456-46bf-b99b-c17304171135\") "
Jul 30 00:46:45.308630 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.308487    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6w5dm\" (UniqueName: \"kubernetes.io/projected/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-kube-api-access-6w5dm\") pod \"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d\" (UID: \"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d\") "
Jul 30 00:46:45.308630 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.308517    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-maas-api-tls\") pod \"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d\" (UID: \"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d\") "
Jul 30 00:46:45.308797 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.308712    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f5b3d9b8-31b5-4fb0-aa82-8a888d69522d" (UID: "f5b3d9b8-31b5-4fb0-aa82-8a888d69522d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:46:45.308961 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.308887    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:46:45.309120 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.309096    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02bcd80d-e456-46bf-b99b-c17304171135-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "02bcd80d-e456-46bf-b99b-c17304171135" (UID: "02bcd80d-e456-46bf-b99b-c17304171135"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:46:45.311578 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.311551    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f5b3d9b8-31b5-4fb0-aa82-8a888d69522d" (UID: "f5b3d9b8-31b5-4fb0-aa82-8a888d69522d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:46:45.311578 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.311567    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bcd80d-e456-46bf-b99b-c17304171135-kube-api-access-xwnsv" (OuterVolumeSpecName: "kube-api-access-xwnsv") pod "02bcd80d-e456-46bf-b99b-c17304171135" (UID: "02bcd80d-e456-46bf-b99b-c17304171135"). InnerVolumeSpecName "kube-api-access-xwnsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:46:45.311804 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.311768    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-kube-api-access-6w5dm" (OuterVolumeSpecName: "kube-api-access-6w5dm") pod "f5b3d9b8-31b5-4fb0-aa82-8a888d69522d" (UID: "f5b3d9b8-31b5-4fb0-aa82-8a888d69522d"). InnerVolumeSpecName "kube-api-access-6w5dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:46:45.325387 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.325364    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj"
Jul 30 00:46:45.325727 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.325706    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b46ef4c5-739c-4bb3-afed-a976fa9983e8" path="/var/lib/kubelet/pods/b46ef4c5-739c-4bb3-afed-a976fa9983e8/volumes"
Jul 30 00:46:45.409221 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.409181    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh9zf\" (UniqueName: \"kubernetes.io/projected/c73153c5-315e-4d14-9d82-f39b524c3906-kube-api-access-hh9zf\") pod \"c73153c5-315e-4d14-9d82-f39b524c3906\" (UID: \"c73153c5-315e-4d14-9d82-f39b524c3906\") "
Jul 30 00:46:45.409410 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.409235    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73153c5-315e-4d14-9d82-f39b524c3906-plugins-config-volume\") pod \"c73153c5-315e-4d14-9d82-f39b524c3906\" (UID: \"c73153c5-315e-4d14-9d82-f39b524c3906\") "
Jul 30 00:46:45.409670 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.409646    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bcd80d-e456-46bf-b99b-c17304171135-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:46:45.409785 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.409673    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwnsv\" (UniqueName: \"kubernetes.io/projected/02bcd80d-e456-46bf-b99b-c17304171135-kube-api-access-xwnsv\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:46:45.409785 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.409691    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w5dm\" (UniqueName: \"kubernetes.io/projected/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-kube-api-access-6w5dm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:46:45.409785 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.409689    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73153c5-315e-4d14-9d82-f39b524c3906-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c73153c5-315e-4d14-9d82-f39b524c3906" (UID: "c73153c5-315e-4d14-9d82-f39b524c3906"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:46:45.409785 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.409706    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:46:45.410213 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.410184    2579 generic.go:356] "Generic (PLEG): container finished" podID="c73153c5-315e-4d14-9d82-f39b524c3906" containerID="7a01f6a52e15f2394811ac161d3a6eec6ac7b23c50eec4f2de9ec5841f683817" exitCode=0
Jul 30 00:46:45.410333 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.410230    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj"
Jul 30 00:46:45.410333 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.410267    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj" event={"ID":"c73153c5-315e-4d14-9d82-f39b524c3906","Type":"ContainerDied","Data":"7a01f6a52e15f2394811ac161d3a6eec6ac7b23c50eec4f2de9ec5841f683817"}
Jul 30 00:46:45.410333 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.410315    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj" event={"ID":"c73153c5-315e-4d14-9d82-f39b524c3906","Type":"ContainerDied","Data":"56acc3b7c1e4d725d68e720f79e3c435e86027ddbee31ed081bf061d629ea83b"}
Jul 30 00:46:45.410501 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.410347    2579 scope.go:117] "RemoveContainer" containerID="7a01f6a52e15f2394811ac161d3a6eec6ac7b23c50eec4f2de9ec5841f683817"
Jul 30 00:46:45.411591 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.411569    2579 generic.go:356] "Generic (PLEG): container finished" podID="f5b3d9b8-31b5-4fb0-aa82-8a888d69522d" containerID="641745ed17e130689c4045577234e73263272441cc5c45bf1c9c77ab8eb80d8e" exitCode=0
Jul 30 00:46:45.411722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.411628    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"
Jul 30 00:46:45.411722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.411659    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx" event={"ID":"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d","Type":"ContainerDied","Data":"641745ed17e130689c4045577234e73263272441cc5c45bf1c9c77ab8eb80d8e"}
Jul 30 00:46:45.411722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.411713    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx" event={"ID":"f5b3d9b8-31b5-4fb0-aa82-8a888d69522d","Type":"ContainerDied","Data":"8b71c9e929fbf93de6e509feca5c5be0343b080798e9055a3c56d35806fd7859"}
Jul 30 00:46:45.411945 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.411861    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73153c5-315e-4d14-9d82-f39b524c3906-kube-api-access-hh9zf" (OuterVolumeSpecName: "kube-api-access-hh9zf") pod "c73153c5-315e-4d14-9d82-f39b524c3906" (UID: "c73153c5-315e-4d14-9d82-f39b524c3906"). InnerVolumeSpecName "kube-api-access-hh9zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:46:45.413084 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.413061    2579 generic.go:356] "Generic (PLEG): container finished" podID="02bcd80d-e456-46bf-b99b-c17304171135" containerID="f4d959b3952d543a133197f95377eb267e50dbf87032f3714c0e72d52c638b03" exitCode=0
Jul 30 00:46:45.413219 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.413095    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76"
Jul 30 00:46:45.413219 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.413130    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76" event={"ID":"02bcd80d-e456-46bf-b99b-c17304171135","Type":"ContainerDied","Data":"f4d959b3952d543a133197f95377eb267e50dbf87032f3714c0e72d52c638b03"}
Jul 30 00:46:45.413219 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.413166    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76" event={"ID":"02bcd80d-e456-46bf-b99b-c17304171135","Type":"ContainerDied","Data":"baa2f2bddceba743a738182a8d0fa7002989113d9750889e9f06df53637511dc"}
Jul 30 00:46:45.420110 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.420094    2579 scope.go:117] "RemoveContainer" containerID="7a01f6a52e15f2394811ac161d3a6eec6ac7b23c50eec4f2de9ec5841f683817"
Jul 30 00:46:45.420380 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:46:45.420363    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a01f6a52e15f2394811ac161d3a6eec6ac7b23c50eec4f2de9ec5841f683817\": container with ID starting with 7a01f6a52e15f2394811ac161d3a6eec6ac7b23c50eec4f2de9ec5841f683817 not found: ID does not exist" containerID="7a01f6a52e15f2394811ac161d3a6eec6ac7b23c50eec4f2de9ec5841f683817"
Jul 30 00:46:45.420436 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.420387    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a01f6a52e15f2394811ac161d3a6eec6ac7b23c50eec4f2de9ec5841f683817"} err="failed to get container status \"7a01f6a52e15f2394811ac161d3a6eec6ac7b23c50eec4f2de9ec5841f683817\": rpc error: code = NotFound desc = could not find container \"7a01f6a52e15f2394811ac161d3a6eec6ac7b23c50eec4f2de9ec5841f683817\": container with ID starting with 7a01f6a52e15f2394811ac161d3a6eec6ac7b23c50eec4f2de9ec5841f683817 not found: ID does not exist"
Jul 30 00:46:45.420436 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.420405    2579 scope.go:117] "RemoveContainer" containerID="641745ed17e130689c4045577234e73263272441cc5c45bf1c9c77ab8eb80d8e"
Jul 30 00:46:45.429945 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.429929    2579 scope.go:117] "RemoveContainer" containerID="641745ed17e130689c4045577234e73263272441cc5c45bf1c9c77ab8eb80d8e"
Jul 30 00:46:45.430193 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:46:45.430177    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"641745ed17e130689c4045577234e73263272441cc5c45bf1c9c77ab8eb80d8e\": container with ID starting with 641745ed17e130689c4045577234e73263272441cc5c45bf1c9c77ab8eb80d8e not found: ID does not exist" containerID="641745ed17e130689c4045577234e73263272441cc5c45bf1c9c77ab8eb80d8e"
Jul 30 00:46:45.430272 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.430197    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641745ed17e130689c4045577234e73263272441cc5c45bf1c9c77ab8eb80d8e"} err="failed to get container status \"641745ed17e130689c4045577234e73263272441cc5c45bf1c9c77ab8eb80d8e\": rpc error: code = NotFound desc = could not find container \"641745ed17e130689c4045577234e73263272441cc5c45bf1c9c77ab8eb80d8e\": container with ID starting with 641745ed17e130689c4045577234e73263272441cc5c45bf1c9c77ab8eb80d8e not found: ID does not exist"
Jul 30 00:46:45.430272 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.430214    2579 scope.go:117] "RemoveContainer" containerID="f4d959b3952d543a133197f95377eb267e50dbf87032f3714c0e72d52c638b03"
Jul 30 00:46:45.434763 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.434743    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76"]
Jul 30 00:46:45.436631 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.436613    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-837664ed-5668ffdf-prz76"]
Jul 30 00:46:45.438347 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.438327    2579 scope.go:117] "RemoveContainer" containerID="f4d959b3952d543a133197f95377eb267e50dbf87032f3714c0e72d52c638b03"
Jul 30 00:46:45.438591 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:46:45.438574    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d959b3952d543a133197f95377eb267e50dbf87032f3714c0e72d52c638b03\": container with ID starting with f4d959b3952d543a133197f95377eb267e50dbf87032f3714c0e72d52c638b03 not found: ID does not exist" containerID="f4d959b3952d543a133197f95377eb267e50dbf87032f3714c0e72d52c638b03"
Jul 30 00:46:45.438656 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.438598    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d959b3952d543a133197f95377eb267e50dbf87032f3714c0e72d52c638b03"} err="failed to get container status \"f4d959b3952d543a133197f95377eb267e50dbf87032f3714c0e72d52c638b03\": rpc error: code = NotFound desc = could not find container \"f4d959b3952d543a133197f95377eb267e50dbf87032f3714c0e72d52c638b03\": container with ID starting with f4d959b3952d543a133197f95377eb267e50dbf87032f3714c0e72d52c638b03 not found: ID does not exist"
Jul 30 00:46:45.447888 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.447868    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"]
Jul 30 00:46:45.454368 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.454346    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-837664ed-db878468c-jtbhx"]
Jul 30 00:46:45.510840 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.510787    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh9zf\" (UniqueName: \"kubernetes.io/projected/c73153c5-315e-4d14-9d82-f39b524c3906-kube-api-access-hh9zf\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:46:45.510840 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.510808    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73153c5-315e-4d14-9d82-f39b524c3906-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:46:45.731519 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.731493    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj"]
Jul 30 00:46:45.735089 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:45.735066    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-837664ed-567b8b4d9b-sffmj"]
Jul 30 00:46:47.322855 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:47.322821    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02bcd80d-e456-46bf-b99b-c17304171135" path="/var/lib/kubelet/pods/02bcd80d-e456-46bf-b99b-c17304171135/volumes"
Jul 30 00:46:47.323231 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:47.323183    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c73153c5-315e-4d14-9d82-f39b524c3906" path="/var/lib/kubelet/pods/c73153c5-315e-4d14-9d82-f39b524c3906/volumes"
Jul 30 00:46:47.323592 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:47.323577    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5b3d9b8-31b5-4fb0-aa82-8a888d69522d" path="/var/lib/kubelet/pods/f5b3d9b8-31b5-4fb0-aa82-8a888d69522d/volumes"
Jul 30 00:46:49.885960 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:49.885923    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"]
Jul 30 00:46:49.886365 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:49.886348    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73153c5-315e-4d14-9d82-f39b524c3906" containerName="payload-pre-processing"
Jul 30 00:46:49.886414 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:49.886368    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73153c5-315e-4d14-9d82-f39b524c3906" containerName="payload-pre-processing"
Jul 30 00:46:49.886414 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:49.886387    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5b3d9b8-31b5-4fb0-aa82-8a888d69522d" containerName="maas-api"
Jul 30 00:46:49.886414 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:49.886392    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5b3d9b8-31b5-4fb0-aa82-8a888d69522d" containerName="maas-api"
Jul 30 00:46:49.886414 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:49.886402    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02bcd80d-e456-46bf-b99b-c17304171135" containerName="payload-processing"
Jul 30 00:46:49.886414 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:49.886407    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="02bcd80d-e456-46bf-b99b-c17304171135" containerName="payload-processing"
Jul 30 00:46:49.886414 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:49.886413    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b46ef4c5-739c-4bb3-afed-a976fa9983e8" containerName="revoke-keys"
Jul 30 00:46:49.886665 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:49.886419    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b46ef4c5-739c-4bb3-afed-a976fa9983e8" containerName="revoke-keys"
Jul 30 00:46:49.886665 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:49.886480    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73153c5-315e-4d14-9d82-f39b524c3906" containerName="payload-pre-processing"
Jul 30 00:46:49.886665 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:49.886491    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5b3d9b8-31b5-4fb0-aa82-8a888d69522d" containerName="maas-api"
Jul 30 00:46:49.886665 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:49.886500    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b46ef4c5-739c-4bb3-afed-a976fa9983e8" containerName="revoke-keys"
Jul 30 00:46:49.886665 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:49.886512    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="02bcd80d-e456-46bf-b99b-c17304171135" containerName="payload-processing"
Jul 30 00:46:49.891645 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:49.891622    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"
Jul 30 00:46:49.894290 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:49.894270    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5213aadc-serving-cert\""
Jul 30 00:46:49.899812 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:49.899791    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"]
Jul 30 00:46:49.950754 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:49.950724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63987e8f-a063-4d77-9755-95bd65d35b4a-openshift-service-ca\") pod \"maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm\" (UID: \"63987e8f-a063-4d77-9755-95bd65d35b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"
Jul 30 00:46:49.950902 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:49.950764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63987e8f-a063-4d77-9755-95bd65d35b4a-maas-api-tls\") pod \"maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm\" (UID: \"63987e8f-a063-4d77-9755-95bd65d35b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"
Jul 30 00:46:49.950902 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:49.950788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8c8c\" (UniqueName: \"kubernetes.io/projected/63987e8f-a063-4d77-9755-95bd65d35b4a-kube-api-access-r8c8c\") pod \"maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm\" (UID: \"63987e8f-a063-4d77-9755-95bd65d35b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"
Jul 30 00:46:50.051679 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.051653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63987e8f-a063-4d77-9755-95bd65d35b4a-openshift-service-ca\") pod \"maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm\" (UID: \"63987e8f-a063-4d77-9755-95bd65d35b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"
Jul 30 00:46:50.051807 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.051691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63987e8f-a063-4d77-9755-95bd65d35b4a-maas-api-tls\") pod \"maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm\" (UID: \"63987e8f-a063-4d77-9755-95bd65d35b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"
Jul 30 00:46:50.051807 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.051714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8c8c\" (UniqueName: \"kubernetes.io/projected/63987e8f-a063-4d77-9755-95bd65d35b4a-kube-api-access-r8c8c\") pod \"maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm\" (UID: \"63987e8f-a063-4d77-9755-95bd65d35b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"
Jul 30 00:46:50.051883 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:46:50.051814    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-serving-cert: secret "maas-api-e2e-ait-5213aadc-serving-cert" not found
Jul 30 00:46:50.051883 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:46:50.051878    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63987e8f-a063-4d77-9755-95bd65d35b4a-maas-api-tls podName:63987e8f-a063-4d77-9755-95bd65d35b4a nodeName:}" failed. No retries permitted until 2026-07-30 00:46:50.551857635 +0000 UTC m=+1735.849124551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63987e8f-a063-4d77-9755-95bd65d35b4a-maas-api-tls") pod "maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm" (UID: "63987e8f-a063-4d77-9755-95bd65d35b4a") : secret "maas-api-e2e-ait-5213aadc-serving-cert" not found
Jul 30 00:46:50.052272 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.052231    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63987e8f-a063-4d77-9755-95bd65d35b4a-openshift-service-ca\") pod \"maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm\" (UID: \"63987e8f-a063-4d77-9755-95bd65d35b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"
Jul 30 00:46:50.062341 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.062316    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8c8c\" (UniqueName: \"kubernetes.io/projected/63987e8f-a063-4d77-9755-95bd65d35b4a-kube-api-access-r8c8c\") pod \"maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm\" (UID: \"63987e8f-a063-4d77-9755-95bd65d35b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"
Jul 30 00:46:50.537559 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.537531    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx"]
Jul 30 00:46:50.540635 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.540613    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx"
Jul 30 00:46:50.543255 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.543216    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5213aadc\""
Jul 30 00:46:50.550802 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.550780    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx"]
Jul 30 00:46:50.555662 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.555642    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63987e8f-a063-4d77-9755-95bd65d35b4a-maas-api-tls\") pod \"maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm\" (UID: \"63987e8f-a063-4d77-9755-95bd65d35b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"
Jul 30 00:46:50.558122 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.558092    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63987e8f-a063-4d77-9755-95bd65d35b4a-maas-api-tls\") pod \"maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm\" (UID: \"63987e8f-a063-4d77-9755-95bd65d35b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"
Jul 30 00:46:50.582758 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.582729    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9"]
Jul 30 00:46:50.585150 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.585134    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9"
Jul 30 00:46:50.592151 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.592120    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5213aadc-dockercfg-p2nvd\""
Jul 30 00:46:50.598633 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.598602    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9"]
Jul 30 00:46:50.656642 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.656608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef61a17-4882-4b9f-be2e-2b9d4e16f81f-plugins-config-volume\") pod \"payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9\" (UID: \"4ef61a17-4882-4b9f-be2e-2b9d4e16f81f\") " pod="openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9"
Jul 30 00:46:50.656779 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.656650    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4b2v\" (UniqueName: \"kubernetes.io/projected/4ef61a17-4882-4b9f-be2e-2b9d4e16f81f-kube-api-access-p4b2v\") pod \"payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9\" (UID: \"4ef61a17-4882-4b9f-be2e-2b9d4e16f81f\") " pod="openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9"
Jul 30 00:46:50.656779 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.656713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/099e332e-06d4-432f-abfa-d96489670ce3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx\" (UID: \"099e332e-06d4-432f-abfa-d96489670ce3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx"
Jul 30 00:46:50.656869 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.656777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxsg2\" (UniqueName: \"kubernetes.io/projected/099e332e-06d4-432f-abfa-d96489670ce3-kube-api-access-vxsg2\") pod \"payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx\" (UID: \"099e332e-06d4-432f-abfa-d96489670ce3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx"
Jul 30 00:46:50.757754 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.757719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef61a17-4882-4b9f-be2e-2b9d4e16f81f-plugins-config-volume\") pod \"payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9\" (UID: \"4ef61a17-4882-4b9f-be2e-2b9d4e16f81f\") " pod="openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9"
Jul 30 00:46:50.757754 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.757751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4b2v\" (UniqueName: \"kubernetes.io/projected/4ef61a17-4882-4b9f-be2e-2b9d4e16f81f-kube-api-access-p4b2v\") pod \"payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9\" (UID: \"4ef61a17-4882-4b9f-be2e-2b9d4e16f81f\") " pod="openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9"
Jul 30 00:46:50.757958 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.757772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/099e332e-06d4-432f-abfa-d96489670ce3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx\" (UID: \"099e332e-06d4-432f-abfa-d96489670ce3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx"
Jul 30 00:46:50.757958 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.757791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxsg2\" (UniqueName: \"kubernetes.io/projected/099e332e-06d4-432f-abfa-d96489670ce3-kube-api-access-vxsg2\") pod \"payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx\" (UID: \"099e332e-06d4-432f-abfa-d96489670ce3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx"
Jul 30 00:46:50.758695 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.758671    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef61a17-4882-4b9f-be2e-2b9d4e16f81f-plugins-config-volume\") pod \"payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9\" (UID: \"4ef61a17-4882-4b9f-be2e-2b9d4e16f81f\") " pod="openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9"
Jul 30 00:46:50.758761 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.758671    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/099e332e-06d4-432f-abfa-d96489670ce3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx\" (UID: \"099e332e-06d4-432f-abfa-d96489670ce3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx"
Jul 30 00:46:50.768160 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.768138    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4b2v\" (UniqueName: \"kubernetes.io/projected/4ef61a17-4882-4b9f-be2e-2b9d4e16f81f-kube-api-access-p4b2v\") pod \"payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9\" (UID: \"4ef61a17-4882-4b9f-be2e-2b9d4e16f81f\") " pod="openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9"
Jul 30 00:46:50.768292 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.768182    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxsg2\" (UniqueName: \"kubernetes.io/projected/099e332e-06d4-432f-abfa-d96489670ce3-kube-api-access-vxsg2\") pod \"payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx\" (UID: \"099e332e-06d4-432f-abfa-d96489670ce3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx"
Jul 30 00:46:50.804280 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.804196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"
Jul 30 00:46:50.852618 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.852580    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx"
Jul 30 00:46:50.903541 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.903509    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9"
Jul 30 00:46:50.953862 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:50.953818    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"]
Jul 30 00:46:50.957606 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:46:50.957574    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63987e8f_a063_4d77_9755_95bd65d35b4a.slice/crio-0a8800becad157cdb7b858caeb0008833a59108d9ad4d43e5d33f7b4b2c92606 WatchSource:0}: Error finding container 0a8800becad157cdb7b858caeb0008833a59108d9ad4d43e5d33f7b4b2c92606: Status 404 returned error can't find the container with id 0a8800becad157cdb7b858caeb0008833a59108d9ad4d43e5d33f7b4b2c92606
Jul 30 00:46:51.014137 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:51.014061    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx"]
Jul 30 00:46:51.015448 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:46:51.015418    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099e332e_06d4_432f_abfa_d96489670ce3.slice/crio-3e6ed53f32995ca35aea8908488d1ee352e480e87bc45788ba31d2d0f2f1da33 WatchSource:0}: Error finding container 3e6ed53f32995ca35aea8908488d1ee352e480e87bc45788ba31d2d0f2f1da33: Status 404 returned error can't find the container with id 3e6ed53f32995ca35aea8908488d1ee352e480e87bc45788ba31d2d0f2f1da33
Jul 30 00:46:51.066168 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:51.066141    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9"]
Jul 30 00:46:51.067527 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:46:51.067321    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef61a17_4882_4b9f_be2e_2b9d4e16f81f.slice/crio-078596a7bab3562fa379fa60cefcc8d76ba514555e34e18fe6ea194cfec33cab WatchSource:0}: Error finding container 078596a7bab3562fa379fa60cefcc8d76ba514555e34e18fe6ea194cfec33cab: Status 404 returned error can't find the container with id 078596a7bab3562fa379fa60cefcc8d76ba514555e34e18fe6ea194cfec33cab
Jul 30 00:46:51.449681 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:51.449652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx" event={"ID":"099e332e-06d4-432f-abfa-d96489670ce3","Type":"ContainerStarted","Data":"ad8db354ab89f005fb08e3246000327e71123f004a105b6ea37982c62870c9fe"}
Jul 30 00:46:51.449810 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:51.449716    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx" event={"ID":"099e332e-06d4-432f-abfa-d96489670ce3","Type":"ContainerStarted","Data":"3e6ed53f32995ca35aea8908488d1ee352e480e87bc45788ba31d2d0f2f1da33"}
Jul 30 00:46:51.449896 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:51.449871    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx"
Jul 30 00:46:51.451868 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:51.451840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9" event={"ID":"4ef61a17-4882-4b9f-be2e-2b9d4e16f81f","Type":"ContainerStarted","Data":"3cf9a484fd1a83a0c6e24947f23a26d262d677ff6e672f42af96dab88d611460"}
Jul 30 00:46:51.451975 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:51.451887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9" event={"ID":"4ef61a17-4882-4b9f-be2e-2b9d4e16f81f","Type":"ContainerStarted","Data":"078596a7bab3562fa379fa60cefcc8d76ba514555e34e18fe6ea194cfec33cab"}
Jul 30 00:46:51.451975 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:51.451913    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9"
Jul 30 00:46:51.453129 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:51.453107    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm" event={"ID":"63987e8f-a063-4d77-9755-95bd65d35b4a","Type":"ContainerStarted","Data":"0a8800becad157cdb7b858caeb0008833a59108d9ad4d43e5d33f7b4b2c92606"}
Jul 30 00:46:51.465869 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:51.465812    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx" podStartSLOduration=1.465790027 podStartE2EDuration="1.465790027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:46:51.464382869 +0000 UTC m=+1736.761649801" watchObservedRunningTime="2026-07-30 00:46:51.465790027 +0000 UTC m=+1736.763056998"
Jul 30 00:46:51.483649 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:51.483603    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9" podStartSLOduration=1.483587779 podStartE2EDuration="1.483587779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:46:51.481464746 +0000 UTC m=+1736.778731678" watchObservedRunningTime="2026-07-30 00:46:51.483587779 +0000 UTC m=+1736.780854711"
Jul 30 00:46:52.460728 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:52.460694    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm" event={"ID":"63987e8f-a063-4d77-9755-95bd65d35b4a","Type":"ContainerStarted","Data":"a721dbc7c461c80b2d6180b9c9f4ab0cc40d763c6a99ff984a37c206defd86d3"}
Jul 30 00:46:52.461162 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:52.460843    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"
Jul 30 00:46:52.480301 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:52.480255    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm" podStartSLOduration=3.007175894 podStartE2EDuration="3.480224643s" totalImagesPullingTime="473.048749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:46:52.477833967 +0000 UTC m=+1737.775100899" watchObservedRunningTime="2026-07-30 00:46:52.480224643 +0000 UTC m=+1737.777491583"
Jul 30 00:46:58.470595 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:46:58.470565    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"
Jul 30 00:47:02.463422 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:02.463394    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9"
Jul 30 00:47:02.463840 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:02.463541    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx"
Jul 30 00:47:14.356000 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:14.355965    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5213aadc-scbhp"]
Jul 30 00:47:14.359656 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:14.359634    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5213aadc-scbhp"
Jul 30 00:47:14.362328 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:14.362309    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v8q7j\""
Jul 30 00:47:14.367809 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:14.367488    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5213aadc-scbhp"]
Jul 30 00:47:14.465903 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:14.465868    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6b23d4-bb33-4c7c-ae20-e0e54696dad9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5213aadc-scbhp\" (UID: \"be6b23d4-bb33-4c7c-ae20-e0e54696dad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5213aadc-scbhp"
Jul 30 00:47:14.567009 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:14.566975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6b23d4-bb33-4c7c-ae20-e0e54696dad9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5213aadc-scbhp\" (UID: \"be6b23d4-bb33-4c7c-ae20-e0e54696dad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5213aadc-scbhp"
Jul 30 00:47:14.567677 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:14.567656    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6b23d4-bb33-4c7c-ae20-e0e54696dad9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5213aadc-scbhp\" (UID: \"be6b23d4-bb33-4c7c-ae20-e0e54696dad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5213aadc-scbhp"
Jul 30 00:47:14.672449 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:14.672422    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5213aadc-scbhp"
Jul 30 00:47:14.802528 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:14.802501    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5213aadc-scbhp"]
Jul 30 00:47:14.805065 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:47:14.805035    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6b23d4_bb33_4c7c_ae20_e0e54696dad9.slice/crio-f4d0b8ed299019251a98814f5a14add9445077dbd47425a2711434fdacf72644 WatchSource:0}: Error finding container f4d0b8ed299019251a98814f5a14add9445077dbd47425a2711434fdacf72644: Status 404 returned error can't find the container with id f4d0b8ed299019251a98814f5a14add9445077dbd47425a2711434fdacf72644
Jul 30 00:47:15.578614 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:15.578583    2579 generic.go:356] "Generic (PLEG): container finished" podID="be6b23d4-bb33-4c7c-ae20-e0e54696dad9" containerID="28afe07d807a8a28bdf2098ec8be620ef80f3fba9de451a1f265d3256f506ea2" exitCode=0
Jul 30 00:47:15.578963 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:15.578671    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5213aadc-scbhp" event={"ID":"be6b23d4-bb33-4c7c-ae20-e0e54696dad9","Type":"ContainerDied","Data":"28afe07d807a8a28bdf2098ec8be620ef80f3fba9de451a1f265d3256f506ea2"}
Jul 30 00:47:15.578963 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:15.578735    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5213aadc-scbhp" event={"ID":"be6b23d4-bb33-4c7c-ae20-e0e54696dad9","Type":"ContainerStarted","Data":"f4d0b8ed299019251a98814f5a14add9445077dbd47425a2711434fdacf72644"}
Jul 30 00:47:16.714621 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:16.714596    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5213aadc-scbhp"
Jul 30 00:47:16.788697 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:16.788673    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6b23d4-bb33-4c7c-ae20-e0e54696dad9-maas-api-service-ca\") pod \"be6b23d4-bb33-4c7c-ae20-e0e54696dad9\" (UID: \"be6b23d4-bb33-4c7c-ae20-e0e54696dad9\") "
Jul 30 00:47:16.789005 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:16.788983    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6b23d4-bb33-4c7c-ae20-e0e54696dad9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be6b23d4-bb33-4c7c-ae20-e0e54696dad9" (UID: "be6b23d4-bb33-4c7c-ae20-e0e54696dad9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:47:16.890078 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:16.890053    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6b23d4-bb33-4c7c-ae20-e0e54696dad9-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:47:17.590572 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:17.590490    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5213aadc-scbhp"
Jul 30 00:47:17.590716 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:17.590497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5213aadc-scbhp" event={"ID":"be6b23d4-bb33-4c7c-ae20-e0e54696dad9","Type":"ContainerDied","Data":"f4d0b8ed299019251a98814f5a14add9445077dbd47425a2711434fdacf72644"}
Jul 30 00:47:17.590716 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:17.590614    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4d0b8ed299019251a98814f5a14add9445077dbd47425a2711434fdacf72644"
Jul 30 00:47:20.344533 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.344496    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"]
Jul 30 00:47:20.344945 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.344745    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm" podUID="63987e8f-a063-4d77-9755-95bd65d35b4a" containerName="maas-api" containerID="cri-o://a721dbc7c461c80b2d6180b9c9f4ab0cc40d763c6a99ff984a37c206defd86d3" gracePeriod=30
Jul 30 00:47:20.399457 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.399426    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9"]
Jul 30 00:47:20.399866 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.399837    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9" podUID="4ef61a17-4882-4b9f-be2e-2b9d4e16f81f" containerName="payload-processing" containerID="cri-o://3cf9a484fd1a83a0c6e24947f23a26d262d677ff6e672f42af96dab88d611460" gracePeriod=30
Jul 30 00:47:20.405375 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.405218    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx"]
Jul 30 00:47:20.405552 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.405523    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx" podUID="099e332e-06d4-432f-abfa-d96489670ce3" containerName="payload-pre-processing" containerID="cri-o://ad8db354ab89f005fb08e3246000327e71123f004a105b6ea37982c62870c9fe" gracePeriod=30
Jul 30 00:47:20.539084 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.539048    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5213aadc-scbhp"]
Jul 30 00:47:20.539345 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.539329    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5213aadc-scbhp"]
Jul 30 00:47:20.608369 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.608340    2579 generic.go:356] "Generic (PLEG): container finished" podID="099e332e-06d4-432f-abfa-d96489670ce3" containerID="ad8db354ab89f005fb08e3246000327e71123f004a105b6ea37982c62870c9fe" exitCode=0
Jul 30 00:47:20.608585 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.608468    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx" event={"ID":"099e332e-06d4-432f-abfa-d96489670ce3","Type":"ContainerDied","Data":"ad8db354ab89f005fb08e3246000327e71123f004a105b6ea37982c62870c9fe"}
Jul 30 00:47:20.610428 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.610367    2579 generic.go:356] "Generic (PLEG): container finished" podID="4ef61a17-4882-4b9f-be2e-2b9d4e16f81f" containerID="3cf9a484fd1a83a0c6e24947f23a26d262d677ff6e672f42af96dab88d611460" exitCode=0
Jul 30 00:47:20.610578 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.610446    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9" event={"ID":"4ef61a17-4882-4b9f-be2e-2b9d4e16f81f","Type":"ContainerDied","Data":"3cf9a484fd1a83a0c6e24947f23a26d262d677ff6e672f42af96dab88d611460"}
Jul 30 00:47:20.612661 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.612636    2579 generic.go:356] "Generic (PLEG): container finished" podID="63987e8f-a063-4d77-9755-95bd65d35b4a" containerID="a721dbc7c461c80b2d6180b9c9f4ab0cc40d763c6a99ff984a37c206defd86d3" exitCode=0
Jul 30 00:47:20.612802 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.612732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm" event={"ID":"63987e8f-a063-4d77-9755-95bd65d35b4a","Type":"ContainerDied","Data":"a721dbc7c461c80b2d6180b9c9f4ab0cc40d763c6a99ff984a37c206defd86d3"}
Jul 30 00:47:20.696175 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.696143    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"
Jul 30 00:47:20.706552 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.706531    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx"
Jul 30 00:47:20.709562 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.709545    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9"
Jul 30 00:47:20.826007 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.825980    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63987e8f-a063-4d77-9755-95bd65d35b4a-maas-api-tls\") pod \"63987e8f-a063-4d77-9755-95bd65d35b4a\" (UID: \"63987e8f-a063-4d77-9755-95bd65d35b4a\") "
Jul 30 00:47:20.826165 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.826020    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxsg2\" (UniqueName: \"kubernetes.io/projected/099e332e-06d4-432f-abfa-d96489670ce3-kube-api-access-vxsg2\") pod \"099e332e-06d4-432f-abfa-d96489670ce3\" (UID: \"099e332e-06d4-432f-abfa-d96489670ce3\") "
Jul 30 00:47:20.826165 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.826067    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/099e332e-06d4-432f-abfa-d96489670ce3-plugins-config-volume\") pod \"099e332e-06d4-432f-abfa-d96489670ce3\" (UID: \"099e332e-06d4-432f-abfa-d96489670ce3\") "
Jul 30 00:47:20.826165 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.826084    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef61a17-4882-4b9f-be2e-2b9d4e16f81f-plugins-config-volume\") pod \"4ef61a17-4882-4b9f-be2e-2b9d4e16f81f\" (UID: \"4ef61a17-4882-4b9f-be2e-2b9d4e16f81f\") "
Jul 30 00:47:20.826165 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.826133    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4b2v\" (UniqueName: \"kubernetes.io/projected/4ef61a17-4882-4b9f-be2e-2b9d4e16f81f-kube-api-access-p4b2v\") pod \"4ef61a17-4882-4b9f-be2e-2b9d4e16f81f\" (UID: \"4ef61a17-4882-4b9f-be2e-2b9d4e16f81f\") "
Jul 30 00:47:20.826165 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.826161    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8c8c\" (UniqueName: \"kubernetes.io/projected/63987e8f-a063-4d77-9755-95bd65d35b4a-kube-api-access-r8c8c\") pod \"63987e8f-a063-4d77-9755-95bd65d35b4a\" (UID: \"63987e8f-a063-4d77-9755-95bd65d35b4a\") "
Jul 30 00:47:20.826472 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.826272    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63987e8f-a063-4d77-9755-95bd65d35b4a-openshift-service-ca\") pod \"63987e8f-a063-4d77-9755-95bd65d35b4a\" (UID: \"63987e8f-a063-4d77-9755-95bd65d35b4a\") "
Jul 30 00:47:20.826572 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.826521    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef61a17-4882-4b9f-be2e-2b9d4e16f81f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ef61a17-4882-4b9f-be2e-2b9d4e16f81f" (UID: "4ef61a17-4882-4b9f-be2e-2b9d4e16f81f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:47:20.826572 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.826557    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099e332e-06d4-432f-abfa-d96489670ce3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "099e332e-06d4-432f-abfa-d96489670ce3" (UID: "099e332e-06d4-432f-abfa-d96489670ce3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:47:20.826702 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.826621    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef61a17-4882-4b9f-be2e-2b9d4e16f81f-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:47:20.826702 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.826627    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63987e8f-a063-4d77-9755-95bd65d35b4a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63987e8f-a063-4d77-9755-95bd65d35b4a" (UID: "63987e8f-a063-4d77-9755-95bd65d35b4a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:47:20.828324 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.828297    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099e332e-06d4-432f-abfa-d96489670ce3-kube-api-access-vxsg2" (OuterVolumeSpecName: "kube-api-access-vxsg2") pod "099e332e-06d4-432f-abfa-d96489670ce3" (UID: "099e332e-06d4-432f-abfa-d96489670ce3"). InnerVolumeSpecName "kube-api-access-vxsg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:47:20.828590 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.828568    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63987e8f-a063-4d77-9755-95bd65d35b4a-kube-api-access-r8c8c" (OuterVolumeSpecName: "kube-api-access-r8c8c") pod "63987e8f-a063-4d77-9755-95bd65d35b4a" (UID: "63987e8f-a063-4d77-9755-95bd65d35b4a"). InnerVolumeSpecName "kube-api-access-r8c8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:47:20.828730 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.828709    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63987e8f-a063-4d77-9755-95bd65d35b4a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63987e8f-a063-4d77-9755-95bd65d35b4a" (UID: "63987e8f-a063-4d77-9755-95bd65d35b4a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:47:20.828830 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.828722    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef61a17-4882-4b9f-be2e-2b9d4e16f81f-kube-api-access-p4b2v" (OuterVolumeSpecName: "kube-api-access-p4b2v") pod "4ef61a17-4882-4b9f-be2e-2b9d4e16f81f" (UID: "4ef61a17-4882-4b9f-be2e-2b9d4e16f81f"). InnerVolumeSpecName "kube-api-access-p4b2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:47:20.927927 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.927891    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/099e332e-06d4-432f-abfa-d96489670ce3-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:47:20.927927 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.927924    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4b2v\" (UniqueName: \"kubernetes.io/projected/4ef61a17-4882-4b9f-be2e-2b9d4e16f81f-kube-api-access-p4b2v\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:47:20.927927 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.927934    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8c8c\" (UniqueName: \"kubernetes.io/projected/63987e8f-a063-4d77-9755-95bd65d35b4a-kube-api-access-r8c8c\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:47:20.928120 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.927943    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63987e8f-a063-4d77-9755-95bd65d35b4a-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:47:20.928120 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.927952    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63987e8f-a063-4d77-9755-95bd65d35b4a-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:47:20.928120 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:20.927960    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxsg2\" (UniqueName: \"kubernetes.io/projected/099e332e-06d4-432f-abfa-d96489670ce3-kube-api-access-vxsg2\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:47:21.323098 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:21.323018    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be6b23d4-bb33-4c7c-ae20-e0e54696dad9" path="/var/lib/kubelet/pods/be6b23d4-bb33-4c7c-ae20-e0e54696dad9/volumes"
Jul 30 00:47:21.618579 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:21.618501    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9"
Jul 30 00:47:21.618997 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:21.618500    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9" event={"ID":"4ef61a17-4882-4b9f-be2e-2b9d4e16f81f","Type":"ContainerDied","Data":"078596a7bab3562fa379fa60cefcc8d76ba514555e34e18fe6ea194cfec33cab"}
Jul 30 00:47:21.618997 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:21.618634    2579 scope.go:117] "RemoveContainer" containerID="3cf9a484fd1a83a0c6e24947f23a26d262d677ff6e672f42af96dab88d611460"
Jul 30 00:47:21.620021 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:21.620001    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"
Jul 30 00:47:21.620021 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:21.620007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm" event={"ID":"63987e8f-a063-4d77-9755-95bd65d35b4a","Type":"ContainerDied","Data":"0a8800becad157cdb7b858caeb0008833a59108d9ad4d43e5d33f7b4b2c92606"}
Jul 30 00:47:21.621584 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:21.621563    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx"
Jul 30 00:47:21.621678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:21.621579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx" event={"ID":"099e332e-06d4-432f-abfa-d96489670ce3","Type":"ContainerDied","Data":"3e6ed53f32995ca35aea8908488d1ee352e480e87bc45788ba31d2d0f2f1da33"}
Jul 30 00:47:21.628534 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:21.628517    2579 scope.go:117] "RemoveContainer" containerID="a721dbc7c461c80b2d6180b9c9f4ab0cc40d763c6a99ff984a37c206defd86d3"
Jul 30 00:47:21.636276 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:21.636234    2579 scope.go:117] "RemoveContainer" containerID="ad8db354ab89f005fb08e3246000327e71123f004a105b6ea37982c62870c9fe"
Jul 30 00:47:21.640524 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:21.640494    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx"]
Jul 30 00:47:21.646363 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:21.646342    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5213aadc-5b5bbc8968-zq2sx"]
Jul 30 00:47:21.658338 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:21.658311    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"]
Jul 30 00:47:21.664369 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:21.664348    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5213aadc-65ff6d4f57-cphrm"]
Jul 30 00:47:21.674679 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:21.674657    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9"]
Jul 30 00:47:21.679025 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:21.679005    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5213aadc-6bbf9bd4d-zh6b9"]
Jul 30 00:47:23.323402 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:23.323370    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="099e332e-06d4-432f-abfa-d96489670ce3" path="/var/lib/kubelet/pods/099e332e-06d4-432f-abfa-d96489670ce3/volumes"
Jul 30 00:47:23.323914 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:23.323895    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ef61a17-4882-4b9f-be2e-2b9d4e16f81f" path="/var/lib/kubelet/pods/4ef61a17-4882-4b9f-be2e-2b9d4e16f81f/volumes"
Jul 30 00:47:23.324339 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:23.324322    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63987e8f-a063-4d77-9755-95bd65d35b4a" path="/var/lib/kubelet/pods/63987e8f-a063-4d77-9755-95bd65d35b4a/volumes"
Jul 30 00:47:25.976848 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:25.976812    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"]
Jul 30 00:47:25.977220 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:25.977205    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ef61a17-4882-4b9f-be2e-2b9d4e16f81f" containerName="payload-processing"
Jul 30 00:47:25.977220 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:25.977221    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ef61a17-4882-4b9f-be2e-2b9d4e16f81f" containerName="payload-processing"
Jul 30 00:47:25.977332 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:25.977269    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="099e332e-06d4-432f-abfa-d96489670ce3" containerName="payload-pre-processing"
Jul 30 00:47:25.977332 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:25.977276    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="099e332e-06d4-432f-abfa-d96489670ce3" containerName="payload-pre-processing"
Jul 30 00:47:25.977332 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:25.977284    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be6b23d4-bb33-4c7c-ae20-e0e54696dad9" containerName="revoke-keys"
Jul 30 00:47:25.977332 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:25.977290    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="be6b23d4-bb33-4c7c-ae20-e0e54696dad9" containerName="revoke-keys"
Jul 30 00:47:25.977332 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:25.977310    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63987e8f-a063-4d77-9755-95bd65d35b4a" containerName="maas-api"
Jul 30 00:47:25.977332 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:25.977315    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="63987e8f-a063-4d77-9755-95bd65d35b4a" containerName="maas-api"
Jul 30 00:47:25.977515 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:25.977373    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="be6b23d4-bb33-4c7c-ae20-e0e54696dad9" containerName="revoke-keys"
Jul 30 00:47:25.977515 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:25.977382    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ef61a17-4882-4b9f-be2e-2b9d4e16f81f" containerName="payload-processing"
Jul 30 00:47:25.977515 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:25.977390    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="63987e8f-a063-4d77-9755-95bd65d35b4a" containerName="maas-api"
Jul 30 00:47:25.977515 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:25.977396    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="099e332e-06d4-432f-abfa-d96489670ce3" containerName="payload-pre-processing"
Jul 30 00:47:25.982198 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:25.982179    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"
Jul 30 00:47:25.985003 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:25.984981    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-06160795-serving-cert\""
Jul 30 00:47:25.991717 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:25.991692    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"]
Jul 30 00:47:26.174793 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.174761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ca22f0-f405-4dbe-a98b-21b6d1b0eb28-openshift-service-ca\") pod \"maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw\" (UID: \"40ca22f0-f405-4dbe-a98b-21b6d1b0eb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"
Jul 30 00:47:26.174793 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.174795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40ca22f0-f405-4dbe-a98b-21b6d1b0eb28-maas-api-tls\") pod \"maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw\" (UID: \"40ca22f0-f405-4dbe-a98b-21b6d1b0eb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"
Jul 30 00:47:26.174988 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.174815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csz4h\" (UniqueName: \"kubernetes.io/projected/40ca22f0-f405-4dbe-a98b-21b6d1b0eb28-kube-api-access-csz4h\") pod \"maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw\" (UID: \"40ca22f0-f405-4dbe-a98b-21b6d1b0eb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"
Jul 30 00:47:26.275429 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.275364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ca22f0-f405-4dbe-a98b-21b6d1b0eb28-openshift-service-ca\") pod \"maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw\" (UID: \"40ca22f0-f405-4dbe-a98b-21b6d1b0eb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"
Jul 30 00:47:26.275429 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.275394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40ca22f0-f405-4dbe-a98b-21b6d1b0eb28-maas-api-tls\") pod \"maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw\" (UID: \"40ca22f0-f405-4dbe-a98b-21b6d1b0eb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"
Jul 30 00:47:26.275429 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.275412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csz4h\" (UniqueName: \"kubernetes.io/projected/40ca22f0-f405-4dbe-a98b-21b6d1b0eb28-kube-api-access-csz4h\") pod \"maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw\" (UID: \"40ca22f0-f405-4dbe-a98b-21b6d1b0eb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"
Jul 30 00:47:26.276002 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.275972    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ca22f0-f405-4dbe-a98b-21b6d1b0eb28-openshift-service-ca\") pod \"maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw\" (UID: \"40ca22f0-f405-4dbe-a98b-21b6d1b0eb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"
Jul 30 00:47:26.277860 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.277837    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40ca22f0-f405-4dbe-a98b-21b6d1b0eb28-maas-api-tls\") pod \"maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw\" (UID: \"40ca22f0-f405-4dbe-a98b-21b6d1b0eb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"
Jul 30 00:47:26.283274 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.283229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csz4h\" (UniqueName: \"kubernetes.io/projected/40ca22f0-f405-4dbe-a98b-21b6d1b0eb28-kube-api-access-csz4h\") pod \"maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw\" (UID: \"40ca22f0-f405-4dbe-a98b-21b6d1b0eb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"
Jul 30 00:47:26.295116 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.295094    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"
Jul 30 00:47:26.436454 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.436421    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"]
Jul 30 00:47:26.439892 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:47:26.439854    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ca22f0_f405_4dbe_a98b_21b6d1b0eb28.slice/crio-b576867f5cceba92b53577ecf75eb40d0654b3ee507c42a7dab16cd43da7e10e WatchSource:0}: Error finding container b576867f5cceba92b53577ecf75eb40d0654b3ee507c42a7dab16cd43da7e10e: Status 404 returned error can't find the container with id b576867f5cceba92b53577ecf75eb40d0654b3ee507c42a7dab16cd43da7e10e
Jul 30 00:47:26.563499 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.563419    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6"]
Jul 30 00:47:26.566910 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.566893    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6"
Jul 30 00:47:26.569831 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.569809    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-06160795-dockercfg-d9vft\""
Jul 30 00:47:26.569921 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.569838    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-06160795\""
Jul 30 00:47:26.577228 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.577206    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6"]
Jul 30 00:47:26.580154 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.580135    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5"]
Jul 30 00:47:26.584022 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.584003    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5"
Jul 30 00:47:26.597676 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.597653    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5"]
Jul 30 00:47:26.650133 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.650101    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw" event={"ID":"40ca22f0-f405-4dbe-a98b-21b6d1b0eb28","Type":"ContainerStarted","Data":"b576867f5cceba92b53577ecf75eb40d0654b3ee507c42a7dab16cd43da7e10e"}
Jul 30 00:47:26.679135 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.679108    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4sgq\" (UniqueName: \"kubernetes.io/projected/af03c2bb-a944-42b2-b85f-64b4fe002ce4-kube-api-access-b4sgq\") pod \"payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6\" (UID: \"af03c2bb-a944-42b2-b85f-64b4fe002ce4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6"
Jul 30 00:47:26.679301 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.679155    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af03c2bb-a944-42b2-b85f-64b4fe002ce4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6\" (UID: \"af03c2bb-a944-42b2-b85f-64b4fe002ce4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6"
Jul 30 00:47:26.780038 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.780008    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4sgq\" (UniqueName: \"kubernetes.io/projected/af03c2bb-a944-42b2-b85f-64b4fe002ce4-kube-api-access-b4sgq\") pod \"payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6\" (UID: \"af03c2bb-a944-42b2-b85f-64b4fe002ce4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6"
Jul 30 00:47:26.780038 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.780043    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af612297-c354-47dd-82e4-42f167b2b9cc-plugins-config-volume\") pod \"payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5\" (UID: \"af612297-c354-47dd-82e4-42f167b2b9cc\") " pod="openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5"
Jul 30 00:47:26.780217 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.780066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af03c2bb-a944-42b2-b85f-64b4fe002ce4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6\" (UID: \"af03c2bb-a944-42b2-b85f-64b4fe002ce4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6"
Jul 30 00:47:26.780217 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.780132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw6f8\" (UniqueName: \"kubernetes.io/projected/af612297-c354-47dd-82e4-42f167b2b9cc-kube-api-access-vw6f8\") pod \"payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5\" (UID: \"af612297-c354-47dd-82e4-42f167b2b9cc\") " pod="openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5"
Jul 30 00:47:26.780764 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.780745    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af03c2bb-a944-42b2-b85f-64b4fe002ce4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6\" (UID: \"af03c2bb-a944-42b2-b85f-64b4fe002ce4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6"
Jul 30 00:47:26.788084 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.788058    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4sgq\" (UniqueName: \"kubernetes.io/projected/af03c2bb-a944-42b2-b85f-64b4fe002ce4-kube-api-access-b4sgq\") pod \"payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6\" (UID: \"af03c2bb-a944-42b2-b85f-64b4fe002ce4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6"
Jul 30 00:47:26.878835 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.878811    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6"
Jul 30 00:47:26.881198 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.881173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw6f8\" (UniqueName: \"kubernetes.io/projected/af612297-c354-47dd-82e4-42f167b2b9cc-kube-api-access-vw6f8\") pod \"payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5\" (UID: \"af612297-c354-47dd-82e4-42f167b2b9cc\") " pod="openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5"
Jul 30 00:47:26.881360 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.881351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af612297-c354-47dd-82e4-42f167b2b9cc-plugins-config-volume\") pod \"payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5\" (UID: \"af612297-c354-47dd-82e4-42f167b2b9cc\") " pod="openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5"
Jul 30 00:47:26.881987 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.881966    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af612297-c354-47dd-82e4-42f167b2b9cc-plugins-config-volume\") pod \"payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5\" (UID: \"af612297-c354-47dd-82e4-42f167b2b9cc\") " pod="openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5"
Jul 30 00:47:26.889465 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.889441    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw6f8\" (UniqueName: \"kubernetes.io/projected/af612297-c354-47dd-82e4-42f167b2b9cc-kube-api-access-vw6f8\") pod \"payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5\" (UID: \"af612297-c354-47dd-82e4-42f167b2b9cc\") " pod="openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5"
Jul 30 00:47:26.894233 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:26.894210    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5"
Jul 30 00:47:27.024940 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:27.024912    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6"]
Jul 30 00:47:27.027296 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:47:27.027219    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf03c2bb_a944_42b2_b85f_64b4fe002ce4.slice/crio-92e6f6b4239b2358d7178e69e8f1646895142dd72d3a0169b21bddac7516279c WatchSource:0}: Error finding container 92e6f6b4239b2358d7178e69e8f1646895142dd72d3a0169b21bddac7516279c: Status 404 returned error can't find the container with id 92e6f6b4239b2358d7178e69e8f1646895142dd72d3a0169b21bddac7516279c
Jul 30 00:47:27.052343 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:27.052315    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5"]
Jul 30 00:47:27.054399 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:47:27.054371    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf612297_c354_47dd_82e4_42f167b2b9cc.slice/crio-cf0c62e019950cef66a973970b79bde286c4164f99c8613c9b7dd8d1c886a0ec WatchSource:0}: Error finding container cf0c62e019950cef66a973970b79bde286c4164f99c8613c9b7dd8d1c886a0ec: Status 404 returned error can't find the container with id cf0c62e019950cef66a973970b79bde286c4164f99c8613c9b7dd8d1c886a0ec
Jul 30 00:47:27.657377 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:27.657339    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5" event={"ID":"af612297-c354-47dd-82e4-42f167b2b9cc","Type":"ContainerStarted","Data":"45eba2b958d2503aee941bbbdba83e07c62e81c99fae7d26de8699d24e55eca7"}
Jul 30 00:47:27.657579 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:27.657400    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5" event={"ID":"af612297-c354-47dd-82e4-42f167b2b9cc","Type":"ContainerStarted","Data":"cf0c62e019950cef66a973970b79bde286c4164f99c8613c9b7dd8d1c886a0ec"}
Jul 30 00:47:27.657579 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:27.657547    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5"
Jul 30 00:47:27.658860 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:27.658837    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6" event={"ID":"af03c2bb-a944-42b2-b85f-64b4fe002ce4","Type":"ContainerStarted","Data":"2ee52464d9c64ff6248c8bbe5e6515671cec4aeafe3d4d6b6a2e59b59601245f"}
Jul 30 00:47:27.658973 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:27.658876    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6" event={"ID":"af03c2bb-a944-42b2-b85f-64b4fe002ce4","Type":"ContainerStarted","Data":"92e6f6b4239b2358d7178e69e8f1646895142dd72d3a0169b21bddac7516279c"}
Jul 30 00:47:27.658973 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:27.658898    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6"
Jul 30 00:47:27.660440 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:27.660417    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw" event={"ID":"40ca22f0-f405-4dbe-a98b-21b6d1b0eb28","Type":"ContainerStarted","Data":"104556f219e45665982882b15c425442235320b5e9b926273ef259dad01ec6a3"}
Jul 30 00:47:27.660544 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:27.660535    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"
Jul 30 00:47:27.676728 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:27.676687    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5" podStartSLOduration=1.676673805 podStartE2EDuration="1.676673805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:47:27.675005867 +0000 UTC m=+1772.972272802" watchObservedRunningTime="2026-07-30 00:47:27.676673805 +0000 UTC m=+1772.973940804"
Jul 30 00:47:27.692005 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:27.691962    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw" podStartSLOduration=2.171839812 podStartE2EDuration="2.691948944s" totalImagesPullingTime="520.109132ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:47:27.690648377 +0000 UTC m=+1772.987915310" watchObservedRunningTime="2026-07-30 00:47:27.691948944 +0000 UTC m=+1772.989215877"
Jul 30 00:47:27.705640 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:27.705603    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6" podStartSLOduration=1.705590451 podStartE2EDuration="1.705590451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:47:27.705391955 +0000 UTC m=+1773.002658888" watchObservedRunningTime="2026-07-30 00:47:27.705590451 +0000 UTC m=+1773.002857406"
Jul 30 00:47:33.671125 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:33.671097    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"
Jul 30 00:47:38.667860 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:38.667829    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6"
Jul 30 00:47:38.668330 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:38.668063    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5"
Jul 30 00:47:49.702417 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:49.702327    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-06160795-lfn6t"]
Jul 30 00:47:49.707214 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:49.707197    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-06160795-lfn6t"
Jul 30 00:47:49.710076 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:49.710041    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v8q7j\""
Jul 30 00:47:49.712932 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:49.712910    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-06160795-lfn6t"]
Jul 30 00:47:49.780236 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:49.780208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a044c6-988b-4947-992b-dc9796a5ff13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-06160795-lfn6t\" (UID: \"14a044c6-988b-4947-992b-dc9796a5ff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-06160795-lfn6t"
Jul 30 00:47:49.881469 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:49.881440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a044c6-988b-4947-992b-dc9796a5ff13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-06160795-lfn6t\" (UID: \"14a044c6-988b-4947-992b-dc9796a5ff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-06160795-lfn6t"
Jul 30 00:47:49.882035 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:49.882013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a044c6-988b-4947-992b-dc9796a5ff13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-06160795-lfn6t\" (UID: \"14a044c6-988b-4947-992b-dc9796a5ff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-06160795-lfn6t"
Jul 30 00:47:50.019939 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:50.019873    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-06160795-lfn6t"
Jul 30 00:47:50.154147 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:50.154122    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-06160795-lfn6t"]
Jul 30 00:47:50.156708 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:47:50.156676    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a044c6_988b_4947_992b_dc9796a5ff13.slice/crio-800c4a8bfd6216b1ad9cc46ac4e93149fd5efc45ae6222ba2a3e74e1cc108341 WatchSource:0}: Error finding container 800c4a8bfd6216b1ad9cc46ac4e93149fd5efc45ae6222ba2a3e74e1cc108341: Status 404 returned error can't find the container with id 800c4a8bfd6216b1ad9cc46ac4e93149fd5efc45ae6222ba2a3e74e1cc108341
Jul 30 00:47:50.779617 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:50.779581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-06160795-lfn6t" event={"ID":"14a044c6-988b-4947-992b-dc9796a5ff13","Type":"ContainerStarted","Data":"c72ece906a17a91c8f80edd8e30fc631e9b8b6e0c9f90b1641067f5f24630e92"}
Jul 30 00:47:50.779964 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:50.779646    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-06160795-lfn6t" event={"ID":"14a044c6-988b-4947-992b-dc9796a5ff13","Type":"ContainerStarted","Data":"800c4a8bfd6216b1ad9cc46ac4e93149fd5efc45ae6222ba2a3e74e1cc108341"}
Jul 30 00:47:50.795749 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:50.795705    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-06160795-lfn6t" podStartSLOduration=1.795688587 podStartE2EDuration="1.795688587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:47:50.793331452 +0000 UTC m=+1796.090598384" watchObservedRunningTime="2026-07-30 00:47:50.795688587 +0000 UTC m=+1796.092955596"
Jul 30 00:47:51.785631 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:51.785599    2579 generic.go:356] "Generic (PLEG): container finished" podID="14a044c6-988b-4947-992b-dc9796a5ff13" containerID="c72ece906a17a91c8f80edd8e30fc631e9b8b6e0c9f90b1641067f5f24630e92" exitCode=0
Jul 30 00:47:51.786017 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:51.785696    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-06160795-lfn6t" event={"ID":"14a044c6-988b-4947-992b-dc9796a5ff13","Type":"ContainerDied","Data":"c72ece906a17a91c8f80edd8e30fc631e9b8b6e0c9f90b1641067f5f24630e92"}
Jul 30 00:47:52.927785 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:52.927762    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-06160795-lfn6t"
Jul 30 00:47:53.005379 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:53.005347    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a044c6-988b-4947-992b-dc9796a5ff13-maas-api-service-ca\") pod \"14a044c6-988b-4947-992b-dc9796a5ff13\" (UID: \"14a044c6-988b-4947-992b-dc9796a5ff13\") "
Jul 30 00:47:53.005671 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:53.005650    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a044c6-988b-4947-992b-dc9796a5ff13-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "14a044c6-988b-4947-992b-dc9796a5ff13" (UID: "14a044c6-988b-4947-992b-dc9796a5ff13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:47:53.106210 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:53.106140    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a044c6-988b-4947-992b-dc9796a5ff13-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:47:53.796342 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:53.796311    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-06160795-lfn6t"
Jul 30 00:47:53.796492 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:53.796355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-06160795-lfn6t" event={"ID":"14a044c6-988b-4947-992b-dc9796a5ff13","Type":"ContainerDied","Data":"800c4a8bfd6216b1ad9cc46ac4e93149fd5efc45ae6222ba2a3e74e1cc108341"}
Jul 30 00:47:53.796492 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:53.796403    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="800c4a8bfd6216b1ad9cc46ac4e93149fd5efc45ae6222ba2a3e74e1cc108341"
Jul 30 00:47:55.284766 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:55.284735    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:47:55.286761 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:55.286741    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:47:56.712649 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:56.712620    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"]
Jul 30 00:47:56.713044 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:56.712844    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw" podUID="40ca22f0-f405-4dbe-a98b-21b6d1b0eb28" containerName="maas-api" containerID="cri-o://104556f219e45665982882b15c425442235320b5e9b926273ef259dad01ec6a3" gracePeriod=30
Jul 30 00:47:56.756978 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:56.756948    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5"]
Jul 30 00:47:56.757167 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:56.757142    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5" podUID="af612297-c354-47dd-82e4-42f167b2b9cc" containerName="payload-processing" containerID="cri-o://45eba2b958d2503aee941bbbdba83e07c62e81c99fae7d26de8699d24e55eca7" gracePeriod=30
Jul 30 00:47:56.762652 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:56.762621    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6"]
Jul 30 00:47:56.762929 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:56.762852    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6" podUID="af03c2bb-a944-42b2-b85f-64b4fe002ce4" containerName="payload-pre-processing" containerID="cri-o://2ee52464d9c64ff6248c8bbe5e6515671cec4aeafe3d4d6b6a2e59b59601245f" gracePeriod=30
Jul 30 00:47:56.885347 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:56.885295    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-06160795-lfn6t"]
Jul 30 00:47:56.888427 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:56.888396    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-06160795-lfn6t"]
Jul 30 00:47:57.037589 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.037539    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5"
Jul 30 00:47:57.042855 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.042827    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"
Jul 30 00:47:57.062410 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.062387    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6"
Jul 30 00:47:57.141708 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.141678    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4sgq\" (UniqueName: \"kubernetes.io/projected/af03c2bb-a944-42b2-b85f-64b4fe002ce4-kube-api-access-b4sgq\") pod \"af03c2bb-a944-42b2-b85f-64b4fe002ce4\" (UID: \"af03c2bb-a944-42b2-b85f-64b4fe002ce4\") "
Jul 30 00:47:57.141865 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.141716    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af612297-c354-47dd-82e4-42f167b2b9cc-plugins-config-volume\") pod \"af612297-c354-47dd-82e4-42f167b2b9cc\" (UID: \"af612297-c354-47dd-82e4-42f167b2b9cc\") "
Jul 30 00:47:57.141865 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.141742    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40ca22f0-f405-4dbe-a98b-21b6d1b0eb28-maas-api-tls\") pod \"40ca22f0-f405-4dbe-a98b-21b6d1b0eb28\" (UID: \"40ca22f0-f405-4dbe-a98b-21b6d1b0eb28\") "
Jul 30 00:47:57.141865 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.141762    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csz4h\" (UniqueName: \"kubernetes.io/projected/40ca22f0-f405-4dbe-a98b-21b6d1b0eb28-kube-api-access-csz4h\") pod \"40ca22f0-f405-4dbe-a98b-21b6d1b0eb28\" (UID: \"40ca22f0-f405-4dbe-a98b-21b6d1b0eb28\") "
Jul 30 00:47:57.141865 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.141806    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ca22f0-f405-4dbe-a98b-21b6d1b0eb28-openshift-service-ca\") pod \"40ca22f0-f405-4dbe-a98b-21b6d1b0eb28\" (UID: \"40ca22f0-f405-4dbe-a98b-21b6d1b0eb28\") "
Jul 30 00:47:57.141865 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.141831    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af03c2bb-a944-42b2-b85f-64b4fe002ce4-plugins-config-volume\") pod \"af03c2bb-a944-42b2-b85f-64b4fe002ce4\" (UID: \"af03c2bb-a944-42b2-b85f-64b4fe002ce4\") "
Jul 30 00:47:57.141865 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.141861    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vw6f8\" (UniqueName: \"kubernetes.io/projected/af612297-c354-47dd-82e4-42f167b2b9cc-kube-api-access-vw6f8\") pod \"af612297-c354-47dd-82e4-42f167b2b9cc\" (UID: \"af612297-c354-47dd-82e4-42f167b2b9cc\") "
Jul 30 00:47:57.142182 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.142142    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af612297-c354-47dd-82e4-42f167b2b9cc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af612297-c354-47dd-82e4-42f167b2b9cc" (UID: "af612297-c354-47dd-82e4-42f167b2b9cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:47:57.142182 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.142165    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40ca22f0-f405-4dbe-a98b-21b6d1b0eb28-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "40ca22f0-f405-4dbe-a98b-21b6d1b0eb28" (UID: "40ca22f0-f405-4dbe-a98b-21b6d1b0eb28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:47:57.142347 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.142233    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af03c2bb-a944-42b2-b85f-64b4fe002ce4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af03c2bb-a944-42b2-b85f-64b4fe002ce4" (UID: "af03c2bb-a944-42b2-b85f-64b4fe002ce4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:47:57.142401 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.142348    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af612297-c354-47dd-82e4-42f167b2b9cc-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:47:57.142401 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.142365    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ca22f0-f405-4dbe-a98b-21b6d1b0eb28-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:47:57.142401 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.142374    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af03c2bb-a944-42b2-b85f-64b4fe002ce4-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:47:57.143984 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.143955    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af03c2bb-a944-42b2-b85f-64b4fe002ce4-kube-api-access-b4sgq" (OuterVolumeSpecName: "kube-api-access-b4sgq") pod "af03c2bb-a944-42b2-b85f-64b4fe002ce4" (UID: "af03c2bb-a944-42b2-b85f-64b4fe002ce4"). InnerVolumeSpecName "kube-api-access-b4sgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:47:57.143984 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.143974    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40ca22f0-f405-4dbe-a98b-21b6d1b0eb28-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "40ca22f0-f405-4dbe-a98b-21b6d1b0eb28" (UID: "40ca22f0-f405-4dbe-a98b-21b6d1b0eb28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:47:57.143984 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.143974    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ca22f0-f405-4dbe-a98b-21b6d1b0eb28-kube-api-access-csz4h" (OuterVolumeSpecName: "kube-api-access-csz4h") pod "40ca22f0-f405-4dbe-a98b-21b6d1b0eb28" (UID: "40ca22f0-f405-4dbe-a98b-21b6d1b0eb28"). InnerVolumeSpecName "kube-api-access-csz4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:47:57.144172 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.144059    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af612297-c354-47dd-82e4-42f167b2b9cc-kube-api-access-vw6f8" (OuterVolumeSpecName: "kube-api-access-vw6f8") pod "af612297-c354-47dd-82e4-42f167b2b9cc" (UID: "af612297-c354-47dd-82e4-42f167b2b9cc"). InnerVolumeSpecName "kube-api-access-vw6f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:47:57.243500 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.243440    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40ca22f0-f405-4dbe-a98b-21b6d1b0eb28-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:47:57.243500 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.243460    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csz4h\" (UniqueName: \"kubernetes.io/projected/40ca22f0-f405-4dbe-a98b-21b6d1b0eb28-kube-api-access-csz4h\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:47:57.243500 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.243470    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw6f8\" (UniqueName: \"kubernetes.io/projected/af612297-c354-47dd-82e4-42f167b2b9cc-kube-api-access-vw6f8\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:47:57.243500 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.243479    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4sgq\" (UniqueName: \"kubernetes.io/projected/af03c2bb-a944-42b2-b85f-64b4fe002ce4-kube-api-access-b4sgq\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:47:57.322798 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.322768    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14a044c6-988b-4947-992b-dc9796a5ff13" path="/var/lib/kubelet/pods/14a044c6-988b-4947-992b-dc9796a5ff13/volumes"
Jul 30 00:47:57.820689 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.820657    2579 generic.go:356] "Generic (PLEG): container finished" podID="af612297-c354-47dd-82e4-42f167b2b9cc" containerID="45eba2b958d2503aee941bbbdba83e07c62e81c99fae7d26de8699d24e55eca7" exitCode=0
Jul 30 00:47:57.821181 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.820706    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5"
Jul 30 00:47:57.821181 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.820736    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5" event={"ID":"af612297-c354-47dd-82e4-42f167b2b9cc","Type":"ContainerDied","Data":"45eba2b958d2503aee941bbbdba83e07c62e81c99fae7d26de8699d24e55eca7"}
Jul 30 00:47:57.821181 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.820793    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5" event={"ID":"af612297-c354-47dd-82e4-42f167b2b9cc","Type":"ContainerDied","Data":"cf0c62e019950cef66a973970b79bde286c4164f99c8613c9b7dd8d1c886a0ec"}
Jul 30 00:47:57.821181 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.820815    2579 scope.go:117] "RemoveContainer" containerID="45eba2b958d2503aee941bbbdba83e07c62e81c99fae7d26de8699d24e55eca7"
Jul 30 00:47:57.822349 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.821985    2579 generic.go:356] "Generic (PLEG): container finished" podID="af03c2bb-a944-42b2-b85f-64b4fe002ce4" containerID="2ee52464d9c64ff6248c8bbe5e6515671cec4aeafe3d4d6b6a2e59b59601245f" exitCode=0
Jul 30 00:47:57.822349 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.822029    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6"
Jul 30 00:47:57.822349 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.822022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6" event={"ID":"af03c2bb-a944-42b2-b85f-64b4fe002ce4","Type":"ContainerDied","Data":"2ee52464d9c64ff6248c8bbe5e6515671cec4aeafe3d4d6b6a2e59b59601245f"}
Jul 30 00:47:57.822349 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.822152    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6" event={"ID":"af03c2bb-a944-42b2-b85f-64b4fe002ce4","Type":"ContainerDied","Data":"92e6f6b4239b2358d7178e69e8f1646895142dd72d3a0169b21bddac7516279c"}
Jul 30 00:47:57.823695 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.823673    2579 generic.go:356] "Generic (PLEG): container finished" podID="40ca22f0-f405-4dbe-a98b-21b6d1b0eb28" containerID="104556f219e45665982882b15c425442235320b5e9b926273ef259dad01ec6a3" exitCode=0
Jul 30 00:47:57.823804 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.823710    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"
Jul 30 00:47:57.823804 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.823733    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw" event={"ID":"40ca22f0-f405-4dbe-a98b-21b6d1b0eb28","Type":"ContainerDied","Data":"104556f219e45665982882b15c425442235320b5e9b926273ef259dad01ec6a3"}
Jul 30 00:47:57.823804 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.823771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw" event={"ID":"40ca22f0-f405-4dbe-a98b-21b6d1b0eb28","Type":"ContainerDied","Data":"b576867f5cceba92b53577ecf75eb40d0654b3ee507c42a7dab16cd43da7e10e"}
Jul 30 00:47:57.830557 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.830543    2579 scope.go:117] "RemoveContainer" containerID="45eba2b958d2503aee941bbbdba83e07c62e81c99fae7d26de8699d24e55eca7"
Jul 30 00:47:57.830812 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:47:57.830795    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45eba2b958d2503aee941bbbdba83e07c62e81c99fae7d26de8699d24e55eca7\": container with ID starting with 45eba2b958d2503aee941bbbdba83e07c62e81c99fae7d26de8699d24e55eca7 not found: ID does not exist" containerID="45eba2b958d2503aee941bbbdba83e07c62e81c99fae7d26de8699d24e55eca7"
Jul 30 00:47:57.830861 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.830820    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45eba2b958d2503aee941bbbdba83e07c62e81c99fae7d26de8699d24e55eca7"} err="failed to get container status \"45eba2b958d2503aee941bbbdba83e07c62e81c99fae7d26de8699d24e55eca7\": rpc error: code = NotFound desc = could not find container \"45eba2b958d2503aee941bbbdba83e07c62e81c99fae7d26de8699d24e55eca7\": container with ID starting with 45eba2b958d2503aee941bbbdba83e07c62e81c99fae7d26de8699d24e55eca7 not found: ID does not exist"
Jul 30 00:47:57.830861 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.830839    2579 scope.go:117] "RemoveContainer" containerID="2ee52464d9c64ff6248c8bbe5e6515671cec4aeafe3d4d6b6a2e59b59601245f"
Jul 30 00:47:57.838773 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.838756    2579 scope.go:117] "RemoveContainer" containerID="2ee52464d9c64ff6248c8bbe5e6515671cec4aeafe3d4d6b6a2e59b59601245f"
Jul 30 00:47:57.839038 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:47:57.839008    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee52464d9c64ff6248c8bbe5e6515671cec4aeafe3d4d6b6a2e59b59601245f\": container with ID starting with 2ee52464d9c64ff6248c8bbe5e6515671cec4aeafe3d4d6b6a2e59b59601245f not found: ID does not exist" containerID="2ee52464d9c64ff6248c8bbe5e6515671cec4aeafe3d4d6b6a2e59b59601245f"
Jul 30 00:47:57.839123 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.839056    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee52464d9c64ff6248c8bbe5e6515671cec4aeafe3d4d6b6a2e59b59601245f"} err="failed to get container status \"2ee52464d9c64ff6248c8bbe5e6515671cec4aeafe3d4d6b6a2e59b59601245f\": rpc error: code = NotFound desc = could not find container \"2ee52464d9c64ff6248c8bbe5e6515671cec4aeafe3d4d6b6a2e59b59601245f\": container with ID starting with 2ee52464d9c64ff6248c8bbe5e6515671cec4aeafe3d4d6b6a2e59b59601245f not found: ID does not exist"
Jul 30 00:47:57.839123 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.839080    2579 scope.go:117] "RemoveContainer" containerID="104556f219e45665982882b15c425442235320b5e9b926273ef259dad01ec6a3"
Jul 30 00:47:57.846649 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.846634    2579 scope.go:117] "RemoveContainer" containerID="104556f219e45665982882b15c425442235320b5e9b926273ef259dad01ec6a3"
Jul 30 00:47:57.846905 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:47:57.846886    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"104556f219e45665982882b15c425442235320b5e9b926273ef259dad01ec6a3\": container with ID starting with 104556f219e45665982882b15c425442235320b5e9b926273ef259dad01ec6a3 not found: ID does not exist" containerID="104556f219e45665982882b15c425442235320b5e9b926273ef259dad01ec6a3"
Jul 30 00:47:57.846951 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.846912    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"104556f219e45665982882b15c425442235320b5e9b926273ef259dad01ec6a3"} err="failed to get container status \"104556f219e45665982882b15c425442235320b5e9b926273ef259dad01ec6a3\": rpc error: code = NotFound desc = could not find container \"104556f219e45665982882b15c425442235320b5e9b926273ef259dad01ec6a3\": container with ID starting with 104556f219e45665982882b15c425442235320b5e9b926273ef259dad01ec6a3 not found: ID does not exist"
Jul 30 00:47:57.850229 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.850209    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5"]
Jul 30 00:47:57.862373 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.862352    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-06160795-6b5cc86766-6vkd5"]
Jul 30 00:47:57.874190 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.874169    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"]
Jul 30 00:47:57.878063 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.878042    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-06160795-6cd658f6d4-cdjkw"]
Jul 30 00:47:57.889020 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.889000    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6"]
Jul 30 00:47:57.893777 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:57.893755    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-06160795-58d9b68568-4x9x6"]
Jul 30 00:47:59.322708 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:59.322676    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40ca22f0-f405-4dbe-a98b-21b6d1b0eb28" path="/var/lib/kubelet/pods/40ca22f0-f405-4dbe-a98b-21b6d1b0eb28/volumes"
Jul 30 00:47:59.323078 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:59.323020    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af03c2bb-a944-42b2-b85f-64b4fe002ce4" path="/var/lib/kubelet/pods/af03c2bb-a944-42b2-b85f-64b4fe002ce4/volumes"
Jul 30 00:47:59.323364 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:47:59.323350    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af612297-c354-47dd-82e4-42f167b2b9cc" path="/var/lib/kubelet/pods/af612297-c354-47dd-82e4-42f167b2b9cc/volumes"
Jul 30 00:49:48.241967 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.241932    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"]
Jul 30 00:49:48.243040 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.242507    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af612297-c354-47dd-82e4-42f167b2b9cc" containerName="payload-processing"
Jul 30 00:49:48.243040 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.242528    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="af612297-c354-47dd-82e4-42f167b2b9cc" containerName="payload-processing"
Jul 30 00:49:48.243040 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.242558    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14a044c6-988b-4947-992b-dc9796a5ff13" containerName="revoke-keys"
Jul 30 00:49:48.243040 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.242567    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="14a044c6-988b-4947-992b-dc9796a5ff13" containerName="revoke-keys"
Jul 30 00:49:48.243040 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.242580    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40ca22f0-f405-4dbe-a98b-21b6d1b0eb28" containerName="maas-api"
Jul 30 00:49:48.243040 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.242587    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="40ca22f0-f405-4dbe-a98b-21b6d1b0eb28" containerName="maas-api"
Jul 30 00:49:48.243040 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.242601    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af03c2bb-a944-42b2-b85f-64b4fe002ce4" containerName="payload-pre-processing"
Jul 30 00:49:48.243040 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.242609    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="af03c2bb-a944-42b2-b85f-64b4fe002ce4" containerName="payload-pre-processing"
Jul 30 00:49:48.243040 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.242701    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="af03c2bb-a944-42b2-b85f-64b4fe002ce4" containerName="payload-pre-processing"
Jul 30 00:49:48.243040 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.242714    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="40ca22f0-f405-4dbe-a98b-21b6d1b0eb28" containerName="maas-api"
Jul 30 00:49:48.243040 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.242722    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="af612297-c354-47dd-82e4-42f167b2b9cc" containerName="payload-processing"
Jul 30 00:49:48.243040 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.242731    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="14a044c6-988b-4947-992b-dc9796a5ff13" containerName="revoke-keys"
Jul 30 00:49:48.245862 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.245838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"
Jul 30 00:49:48.248580 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.248551    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-819bad55-serving-cert\""
Jul 30 00:49:48.256394 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.256370    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"]
Jul 30 00:49:48.404198 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.404164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d5tq\" (UniqueName: \"kubernetes.io/projected/cdcd0dec-643d-4a69-a150-6e3ee121eca0-kube-api-access-7d5tq\") pod \"maas-api-e2e-mt-819bad55-58b8895b67-2w6zs\" (UID: \"cdcd0dec-643d-4a69-a150-6e3ee121eca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"
Jul 30 00:49:48.404402 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.404206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcd0dec-643d-4a69-a150-6e3ee121eca0-openshift-service-ca\") pod \"maas-api-e2e-mt-819bad55-58b8895b67-2w6zs\" (UID: \"cdcd0dec-643d-4a69-a150-6e3ee121eca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"
Jul 30 00:49:48.404402 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.404238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcd0dec-643d-4a69-a150-6e3ee121eca0-maas-api-tls\") pod \"maas-api-e2e-mt-819bad55-58b8895b67-2w6zs\" (UID: \"cdcd0dec-643d-4a69-a150-6e3ee121eca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"
Jul 30 00:49:48.505730 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.505652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d5tq\" (UniqueName: \"kubernetes.io/projected/cdcd0dec-643d-4a69-a150-6e3ee121eca0-kube-api-access-7d5tq\") pod \"maas-api-e2e-mt-819bad55-58b8895b67-2w6zs\" (UID: \"cdcd0dec-643d-4a69-a150-6e3ee121eca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"
Jul 30 00:49:48.505730 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.505688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcd0dec-643d-4a69-a150-6e3ee121eca0-openshift-service-ca\") pod \"maas-api-e2e-mt-819bad55-58b8895b67-2w6zs\" (UID: \"cdcd0dec-643d-4a69-a150-6e3ee121eca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"
Jul 30 00:49:48.505730 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.505720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcd0dec-643d-4a69-a150-6e3ee121eca0-maas-api-tls\") pod \"maas-api-e2e-mt-819bad55-58b8895b67-2w6zs\" (UID: \"cdcd0dec-643d-4a69-a150-6e3ee121eca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"
Jul 30 00:49:48.506000 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:49:48.505884    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-serving-cert: secret "maas-api-e2e-mt-819bad55-serving-cert" not found
Jul 30 00:49:48.506000 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:49:48.505972    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdcd0dec-643d-4a69-a150-6e3ee121eca0-maas-api-tls podName:cdcd0dec-643d-4a69-a150-6e3ee121eca0 nodeName:}" failed. No retries permitted until 2026-07-30 00:49:49.005948536 +0000 UTC m=+1914.303215454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cdcd0dec-643d-4a69-a150-6e3ee121eca0-maas-api-tls") pod "maas-api-e2e-mt-819bad55-58b8895b67-2w6zs" (UID: "cdcd0dec-643d-4a69-a150-6e3ee121eca0") : secret "maas-api-e2e-mt-819bad55-serving-cert" not found
Jul 30 00:49:48.506432 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.506413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcd0dec-643d-4a69-a150-6e3ee121eca0-openshift-service-ca\") pod \"maas-api-e2e-mt-819bad55-58b8895b67-2w6zs\" (UID: \"cdcd0dec-643d-4a69-a150-6e3ee121eca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"
Jul 30 00:49:48.515142 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.515115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d5tq\" (UniqueName: \"kubernetes.io/projected/cdcd0dec-643d-4a69-a150-6e3ee121eca0-kube-api-access-7d5tq\") pod \"maas-api-e2e-mt-819bad55-58b8895b67-2w6zs\" (UID: \"cdcd0dec-643d-4a69-a150-6e3ee121eca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"
Jul 30 00:49:48.994555 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.994521    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz"]
Jul 30 00:49:48.998051 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:48.998034    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz"
Jul 30 00:49:49.000745 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.000723    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-819bad55-dockercfg-gh9p5\""
Jul 30 00:49:49.000874 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.000728    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-819bad55\""
Jul 30 00:49:49.008196 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.008177    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz"]
Jul 30 00:49:49.010323 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.010281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcd0dec-643d-4a69-a150-6e3ee121eca0-maas-api-tls\") pod \"maas-api-e2e-mt-819bad55-58b8895b67-2w6zs\" (UID: \"cdcd0dec-643d-4a69-a150-6e3ee121eca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"
Jul 30 00:49:49.012775 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.012748    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcd0dec-643d-4a69-a150-6e3ee121eca0-maas-api-tls\") pod \"maas-api-e2e-mt-819bad55-58b8895b67-2w6zs\" (UID: \"cdcd0dec-643d-4a69-a150-6e3ee121eca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"
Jul 30 00:49:49.020815 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.020784    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9"]
Jul 30 00:49:49.024473 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.024454    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9"
Jul 30 00:49:49.033979 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.033957    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9"]
Jul 30 00:49:49.111908 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.111876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c020def-1cb7-4587-929a-da99d5b802f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz\" (UID: \"1c020def-1cb7-4587-929a-da99d5b802f1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz"
Jul 30 00:49:49.112123 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.112104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm5gf\" (UniqueName: \"kubernetes.io/projected/1c020def-1cb7-4587-929a-da99d5b802f1-kube-api-access-xm5gf\") pod \"payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz\" (UID: \"1c020def-1cb7-4587-929a-da99d5b802f1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz"
Jul 30 00:49:49.157847 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.157818    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"
Jul 30 00:49:49.213685 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.213655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b6a154-4679-4c43-9dfe-9919e2bb1439-plugins-config-volume\") pod \"payload-processing-e2e-mt-819bad55-c85866458-4psp9\" (UID: \"94b6a154-4679-4c43-9dfe-9919e2bb1439\") " pod="openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9"
Jul 30 00:49:49.213838 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.213718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm5gf\" (UniqueName: \"kubernetes.io/projected/1c020def-1cb7-4587-929a-da99d5b802f1-kube-api-access-xm5gf\") pod \"payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz\" (UID: \"1c020def-1cb7-4587-929a-da99d5b802f1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz"
Jul 30 00:49:49.213838 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.213773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c020def-1cb7-4587-929a-da99d5b802f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz\" (UID: \"1c020def-1cb7-4587-929a-da99d5b802f1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz"
Jul 30 00:49:49.213838 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.213799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlmmg\" (UniqueName: \"kubernetes.io/projected/94b6a154-4679-4c43-9dfe-9919e2bb1439-kube-api-access-tlmmg\") pod \"payload-processing-e2e-mt-819bad55-c85866458-4psp9\" (UID: \"94b6a154-4679-4c43-9dfe-9919e2bb1439\") " pod="openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9"
Jul 30 00:49:49.214540 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.214517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c020def-1cb7-4587-929a-da99d5b802f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz\" (UID: \"1c020def-1cb7-4587-929a-da99d5b802f1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz"
Jul 30 00:49:49.223322 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.223296    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm5gf\" (UniqueName: \"kubernetes.io/projected/1c020def-1cb7-4587-929a-da99d5b802f1-kube-api-access-xm5gf\") pod \"payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz\" (UID: \"1c020def-1cb7-4587-929a-da99d5b802f1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz"
Jul 30 00:49:49.294214 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.294190    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"]
Jul 30 00:49:49.296689 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:49:49.296659    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdcd0dec_643d_4a69_a150_6e3ee121eca0.slice/crio-befed6ef22d51f338aff64ead377543df1d91acb3eda235f460ae5a9261fb1be WatchSource:0}: Error finding container befed6ef22d51f338aff64ead377543df1d91acb3eda235f460ae5a9261fb1be: Status 404 returned error can't find the container with id befed6ef22d51f338aff64ead377543df1d91acb3eda235f460ae5a9261fb1be
Jul 30 00:49:49.309929 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.309904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz"
Jul 30 00:49:49.314977 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.314954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b6a154-4679-4c43-9dfe-9919e2bb1439-plugins-config-volume\") pod \"payload-processing-e2e-mt-819bad55-c85866458-4psp9\" (UID: \"94b6a154-4679-4c43-9dfe-9919e2bb1439\") " pod="openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9"
Jul 30 00:49:49.315087 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.314995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlmmg\" (UniqueName: \"kubernetes.io/projected/94b6a154-4679-4c43-9dfe-9919e2bb1439-kube-api-access-tlmmg\") pod \"payload-processing-e2e-mt-819bad55-c85866458-4psp9\" (UID: \"94b6a154-4679-4c43-9dfe-9919e2bb1439\") " pod="openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9"
Jul 30 00:49:49.315664 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.315641    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b6a154-4679-4c43-9dfe-9919e2bb1439-plugins-config-volume\") pod \"payload-processing-e2e-mt-819bad55-c85866458-4psp9\" (UID: \"94b6a154-4679-4c43-9dfe-9919e2bb1439\") " pod="openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9"
Jul 30 00:49:49.323809 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.323783    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlmmg\" (UniqueName: \"kubernetes.io/projected/94b6a154-4679-4c43-9dfe-9919e2bb1439-kube-api-access-tlmmg\") pod \"payload-processing-e2e-mt-819bad55-c85866458-4psp9\" (UID: \"94b6a154-4679-4c43-9dfe-9919e2bb1439\") " pod="openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9"
Jul 30 00:49:49.336365 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.336345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9"
Jul 30 00:49:49.367344 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.367295    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs" event={"ID":"cdcd0dec-643d-4a69-a150-6e3ee121eca0","Type":"ContainerStarted","Data":"befed6ef22d51f338aff64ead377543df1d91acb3eda235f460ae5a9261fb1be"}
Jul 30 00:49:49.463420 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.463100    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz"]
Jul 30 00:49:49.467021 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:49:49.466992    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c020def_1cb7_4587_929a_da99d5b802f1.slice/crio-4305429f542ffbf3fdffa42d4915c86bd7b79246bc9ba1e167a9309eecef755c WatchSource:0}: Error finding container 4305429f542ffbf3fdffa42d4915c86bd7b79246bc9ba1e167a9309eecef755c: Status 404 returned error can't find the container with id 4305429f542ffbf3fdffa42d4915c86bd7b79246bc9ba1e167a9309eecef755c
Jul 30 00:49:49.486211 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:49.486189    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9"]
Jul 30 00:49:49.489067 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:49:49.489037    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b6a154_4679_4c43_9dfe_9919e2bb1439.slice/crio-8c7f612843bbb4d0a2cbd677a98ab6c9ca7ead63f1d8d99276198cfdc9df0655 WatchSource:0}: Error finding container 8c7f612843bbb4d0a2cbd677a98ab6c9ca7ead63f1d8d99276198cfdc9df0655: Status 404 returned error can't find the container with id 8c7f612843bbb4d0a2cbd677a98ab6c9ca7ead63f1d8d99276198cfdc9df0655
Jul 30 00:49:50.375573 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:50.375535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9" event={"ID":"94b6a154-4679-4c43-9dfe-9919e2bb1439","Type":"ContainerStarted","Data":"37fc879e2093a92a12995b998c578c90521c59028b59de2b56258f0336e83760"}
Jul 30 00:49:50.376071 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:50.375598    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9" event={"ID":"94b6a154-4679-4c43-9dfe-9919e2bb1439","Type":"ContainerStarted","Data":"8c7f612843bbb4d0a2cbd677a98ab6c9ca7ead63f1d8d99276198cfdc9df0655"}
Jul 30 00:49:50.376071 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:50.375660    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9"
Jul 30 00:49:50.377436 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:50.377394    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz" event={"ID":"1c020def-1cb7-4587-929a-da99d5b802f1","Type":"ContainerStarted","Data":"11770a7aa2f1a4c444586d6cf49a0da26512cfb7fc9d4d2318b56bcc03d27a29"}
Jul 30 00:49:50.377576 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:50.377462    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz" event={"ID":"1c020def-1cb7-4587-929a-da99d5b802f1","Type":"ContainerStarted","Data":"4305429f542ffbf3fdffa42d4915c86bd7b79246bc9ba1e167a9309eecef755c"}
Jul 30 00:49:50.377576 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:50.377495    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz"
Jul 30 00:49:50.394622 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:50.394577    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9" podStartSLOduration=1.394562327 podStartE2EDuration="1.394562327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:49:50.393717583 +0000 UTC m=+1915.690984517" watchObservedRunningTime="2026-07-30 00:49:50.394562327 +0000 UTC m=+1915.691829259"
Jul 30 00:49:50.410258 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:50.410197    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz" podStartSLOduration=2.410185468 podStartE2EDuration="2.410185468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:49:50.408756213 +0000 UTC m=+1915.706023139" watchObservedRunningTime="2026-07-30 00:49:50.410185468 +0000 UTC m=+1915.707452377"
Jul 30 00:49:51.383627 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:51.383591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs" event={"ID":"cdcd0dec-643d-4a69-a150-6e3ee121eca0","Type":"ContainerStarted","Data":"b6fc9a34aeff193715461c9d47dfdcfb03ecf114641368918306f91f849ef50c"}
Jul 30 00:49:51.400675 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:51.400623    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs" podStartSLOduration=1.654012721 podStartE2EDuration="3.400607297s" totalImagesPullingTime="1.746594576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:49:51.400102464 +0000 UTC m=+1916.697369396" watchObservedRunningTime="2026-07-30 00:49:51.400607297 +0000 UTC m=+1916.697874228"
Jul 30 00:49:52.388542 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:52.388513    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"
Jul 30 00:49:58.399435 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:49:58.399406    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"
Jul 30 00:50:01.386170 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:01.386138    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz"
Jul 30 00:50:01.386573 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:01.386214    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9"
Jul 30 00:50:19.220373 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:19.220339    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-819bad55-4ngpz"]
Jul 30 00:50:19.223742 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:19.223719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-819bad55-4ngpz"
Jul 30 00:50:19.226992 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:19.226970    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v8q7j\""
Jul 30 00:50:19.241545 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:19.241520    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-819bad55-4ngpz"]
Jul 30 00:50:19.362614 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:19.362582    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-819bad55-4ngpz\" (UID: \"4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-819bad55-4ngpz"
Jul 30 00:50:19.463484 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:19.463455    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-819bad55-4ngpz\" (UID: \"4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-819bad55-4ngpz"
Jul 30 00:50:19.464124 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:19.464103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-819bad55-4ngpz\" (UID: \"4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-819bad55-4ngpz"
Jul 30 00:50:19.533371 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:19.533310    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-819bad55-4ngpz"
Jul 30 00:50:19.667077 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:19.667050    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-819bad55-4ngpz"]
Jul 30 00:50:19.669904 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:50:19.669873    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c95c8e6_e8cb_4fa7_9730_4dbe9079eccf.slice/crio-317e840bd17a8cae23b163d7da1cdf86229fb0e8075f6da6d2f03ee35fffbe2c WatchSource:0}: Error finding container 317e840bd17a8cae23b163d7da1cdf86229fb0e8075f6da6d2f03ee35fffbe2c: Status 404 returned error can't find the container with id 317e840bd17a8cae23b163d7da1cdf86229fb0e8075f6da6d2f03ee35fffbe2c
Jul 30 00:50:20.528085 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:20.527999    2579 generic.go:356] "Generic (PLEG): container finished" podID="4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf" containerID="9469025ecbea3d45eecec916847757b96febaedd8fda5c52ffe6a4ef41b3c550" exitCode=0
Jul 30 00:50:20.528085 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:20.528041    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-819bad55-4ngpz" event={"ID":"4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf","Type":"ContainerDied","Data":"9469025ecbea3d45eecec916847757b96febaedd8fda5c52ffe6a4ef41b3c550"}
Jul 30 00:50:20.528508 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:20.528095    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-819bad55-4ngpz" event={"ID":"4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf","Type":"ContainerStarted","Data":"317e840bd17a8cae23b163d7da1cdf86229fb0e8075f6da6d2f03ee35fffbe2c"}
Jul 30 00:50:21.666151 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:21.666129    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-819bad55-4ngpz"
Jul 30 00:50:21.786725 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:21.786643    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf-maas-api-service-ca\") pod \"4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf\" (UID: \"4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf\") "
Jul 30 00:50:21.786977 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:21.786954    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf" (UID: "4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:50:21.887739 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:21.887702    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:50:22.546305 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:22.546279    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-819bad55-4ngpz"
Jul 30 00:50:22.546487 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:22.546269    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-819bad55-4ngpz" event={"ID":"4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf","Type":"ContainerDied","Data":"317e840bd17a8cae23b163d7da1cdf86229fb0e8075f6da6d2f03ee35fffbe2c"}
Jul 30 00:50:22.546487 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:22.546416    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="317e840bd17a8cae23b163d7da1cdf86229fb0e8075f6da6d2f03ee35fffbe2c"
Jul 30 00:50:29.582807 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:29.582775    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"]
Jul 30 00:50:29.583370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:29.582999    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs" podUID="cdcd0dec-643d-4a69-a150-6e3ee121eca0" containerName="maas-api" containerID="cri-o://b6fc9a34aeff193715461c9d47dfdcfb03ecf114641368918306f91f849ef50c" gracePeriod=30
Jul 30 00:50:29.721975 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:29.721928    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz"]
Jul 30 00:50:29.722494 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:29.722442    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz" podUID="1c020def-1cb7-4587-929a-da99d5b802f1" containerName="payload-pre-processing" containerID="cri-o://11770a7aa2f1a4c444586d6cf49a0da26512cfb7fc9d4d2318b56bcc03d27a29" gracePeriod=30
Jul 30 00:50:29.724013 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:29.723984    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9"]
Jul 30 00:50:29.724224 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:29.724198    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9" podUID="94b6a154-4679-4c43-9dfe-9919e2bb1439" containerName="payload-processing" containerID="cri-o://37fc879e2093a92a12995b998c578c90521c59028b59de2b56258f0336e83760" gracePeriod=30
Jul 30 00:50:29.850698 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:29.850661    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-819bad55-4ngpz"]
Jul 30 00:50:29.853178 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:29.853150    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-819bad55-4ngpz"]
Jul 30 00:50:29.880003 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:29.879971    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"
Jul 30 00:50:29.959711 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:29.959685    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcd0dec-643d-4a69-a150-6e3ee121eca0-openshift-service-ca\") pod \"cdcd0dec-643d-4a69-a150-6e3ee121eca0\" (UID: \"cdcd0dec-643d-4a69-a150-6e3ee121eca0\") "
Jul 30 00:50:29.959857 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:29.959759    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7d5tq\" (UniqueName: \"kubernetes.io/projected/cdcd0dec-643d-4a69-a150-6e3ee121eca0-kube-api-access-7d5tq\") pod \"cdcd0dec-643d-4a69-a150-6e3ee121eca0\" (UID: \"cdcd0dec-643d-4a69-a150-6e3ee121eca0\") "
Jul 30 00:50:29.959920 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:29.959907    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcd0dec-643d-4a69-a150-6e3ee121eca0-maas-api-tls\") pod \"cdcd0dec-643d-4a69-a150-6e3ee121eca0\" (UID: \"cdcd0dec-643d-4a69-a150-6e3ee121eca0\") "
Jul 30 00:50:29.960160 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:29.960124    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdcd0dec-643d-4a69-a150-6e3ee121eca0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cdcd0dec-643d-4a69-a150-6e3ee121eca0" (UID: "cdcd0dec-643d-4a69-a150-6e3ee121eca0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:50:29.960425 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:29.960379    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcd0dec-643d-4a69-a150-6e3ee121eca0-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:50:29.962104 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:29.962075    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdcd0dec-643d-4a69-a150-6e3ee121eca0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cdcd0dec-643d-4a69-a150-6e3ee121eca0" (UID: "cdcd0dec-643d-4a69-a150-6e3ee121eca0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:50:29.962233 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:29.962214    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdcd0dec-643d-4a69-a150-6e3ee121eca0-kube-api-access-7d5tq" (OuterVolumeSpecName: "kube-api-access-7d5tq") pod "cdcd0dec-643d-4a69-a150-6e3ee121eca0" (UID: "cdcd0dec-643d-4a69-a150-6e3ee121eca0"). InnerVolumeSpecName "kube-api-access-7d5tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:50:29.972204 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:29.972181    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz"
Jul 30 00:50:30.010033 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.010008    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9"
Jul 30 00:50:30.061616 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.061583    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c020def-1cb7-4587-929a-da99d5b802f1-plugins-config-volume\") pod \"1c020def-1cb7-4587-929a-da99d5b802f1\" (UID: \"1c020def-1cb7-4587-929a-da99d5b802f1\") "
Jul 30 00:50:30.061784 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.061645    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm5gf\" (UniqueName: \"kubernetes.io/projected/1c020def-1cb7-4587-929a-da99d5b802f1-kube-api-access-xm5gf\") pod \"1c020def-1cb7-4587-929a-da99d5b802f1\" (UID: \"1c020def-1cb7-4587-929a-da99d5b802f1\") "
Jul 30 00:50:30.061985 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.061967    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcd0dec-643d-4a69-a150-6e3ee121eca0-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:50:30.062042 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.061983    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c020def-1cb7-4587-929a-da99d5b802f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c020def-1cb7-4587-929a-da99d5b802f1" (UID: "1c020def-1cb7-4587-929a-da99d5b802f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:50:30.062042 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.061991    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d5tq\" (UniqueName: \"kubernetes.io/projected/cdcd0dec-643d-4a69-a150-6e3ee121eca0-kube-api-access-7d5tq\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:50:30.063707 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.063670    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c020def-1cb7-4587-929a-da99d5b802f1-kube-api-access-xm5gf" (OuterVolumeSpecName: "kube-api-access-xm5gf") pod "1c020def-1cb7-4587-929a-da99d5b802f1" (UID: "1c020def-1cb7-4587-929a-da99d5b802f1"). InnerVolumeSpecName "kube-api-access-xm5gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:50:30.162539 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.162517    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b6a154-4679-4c43-9dfe-9919e2bb1439-plugins-config-volume\") pod \"94b6a154-4679-4c43-9dfe-9919e2bb1439\" (UID: \"94b6a154-4679-4c43-9dfe-9919e2bb1439\") "
Jul 30 00:50:30.162685 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.162567    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlmmg\" (UniqueName: \"kubernetes.io/projected/94b6a154-4679-4c43-9dfe-9919e2bb1439-kube-api-access-tlmmg\") pod \"94b6a154-4679-4c43-9dfe-9919e2bb1439\" (UID: \"94b6a154-4679-4c43-9dfe-9919e2bb1439\") "
Jul 30 00:50:30.162760 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.162744    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c020def-1cb7-4587-929a-da99d5b802f1-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:50:30.162760 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.162755    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm5gf\" (UniqueName: \"kubernetes.io/projected/1c020def-1cb7-4587-929a-da99d5b802f1-kube-api-access-xm5gf\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:50:30.162874 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.162845    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b6a154-4679-4c43-9dfe-9919e2bb1439-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "94b6a154-4679-4c43-9dfe-9919e2bb1439" (UID: "94b6a154-4679-4c43-9dfe-9919e2bb1439"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:50:30.164710 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.164686    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b6a154-4679-4c43-9dfe-9919e2bb1439-kube-api-access-tlmmg" (OuterVolumeSpecName: "kube-api-access-tlmmg") pod "94b6a154-4679-4c43-9dfe-9919e2bb1439" (UID: "94b6a154-4679-4c43-9dfe-9919e2bb1439"). InnerVolumeSpecName "kube-api-access-tlmmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:50:30.263465 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.263438    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b6a154-4679-4c43-9dfe-9919e2bb1439-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:50:30.263465 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.263460    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlmmg\" (UniqueName: \"kubernetes.io/projected/94b6a154-4679-4c43-9dfe-9919e2bb1439-kube-api-access-tlmmg\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:50:30.589355 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.589269    2579 generic.go:356] "Generic (PLEG): container finished" podID="94b6a154-4679-4c43-9dfe-9919e2bb1439" containerID="37fc879e2093a92a12995b998c578c90521c59028b59de2b56258f0336e83760" exitCode=0
Jul 30 00:50:30.589355 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.589320    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9"
Jul 30 00:50:30.589869 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.589365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9" event={"ID":"94b6a154-4679-4c43-9dfe-9919e2bb1439","Type":"ContainerDied","Data":"37fc879e2093a92a12995b998c578c90521c59028b59de2b56258f0336e83760"}
Jul 30 00:50:30.589869 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.589421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9" event={"ID":"94b6a154-4679-4c43-9dfe-9919e2bb1439","Type":"ContainerDied","Data":"8c7f612843bbb4d0a2cbd677a98ab6c9ca7ead63f1d8d99276198cfdc9df0655"}
Jul 30 00:50:30.589869 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.589449    2579 scope.go:117] "RemoveContainer" containerID="37fc879e2093a92a12995b998c578c90521c59028b59de2b56258f0336e83760"
Jul 30 00:50:30.590936 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.590808    2579 generic.go:356] "Generic (PLEG): container finished" podID="1c020def-1cb7-4587-929a-da99d5b802f1" containerID="11770a7aa2f1a4c444586d6cf49a0da26512cfb7fc9d4d2318b56bcc03d27a29" exitCode=0
Jul 30 00:50:30.590936 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.590874    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz"
Jul 30 00:50:30.590936 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.590913    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz" event={"ID":"1c020def-1cb7-4587-929a-da99d5b802f1","Type":"ContainerDied","Data":"11770a7aa2f1a4c444586d6cf49a0da26512cfb7fc9d4d2318b56bcc03d27a29"}
Jul 30 00:50:30.591109 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.590958    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz" event={"ID":"1c020def-1cb7-4587-929a-da99d5b802f1","Type":"ContainerDied","Data":"4305429f542ffbf3fdffa42d4915c86bd7b79246bc9ba1e167a9309eecef755c"}
Jul 30 00:50:30.592440 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.592363    2579 generic.go:356] "Generic (PLEG): container finished" podID="cdcd0dec-643d-4a69-a150-6e3ee121eca0" containerID="b6fc9a34aeff193715461c9d47dfdcfb03ecf114641368918306f91f849ef50c" exitCode=0
Jul 30 00:50:30.592440 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.592392    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs" event={"ID":"cdcd0dec-643d-4a69-a150-6e3ee121eca0","Type":"ContainerDied","Data":"b6fc9a34aeff193715461c9d47dfdcfb03ecf114641368918306f91f849ef50c"}
Jul 30 00:50:30.592612 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.592459    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs" event={"ID":"cdcd0dec-643d-4a69-a150-6e3ee121eca0","Type":"ContainerDied","Data":"befed6ef22d51f338aff64ead377543df1d91acb3eda235f460ae5a9261fb1be"}
Jul 30 00:50:30.592612 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.592405    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"
Jul 30 00:50:30.612714 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.612696    2579 scope.go:117] "RemoveContainer" containerID="37fc879e2093a92a12995b998c578c90521c59028b59de2b56258f0336e83760"
Jul 30 00:50:30.613029 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:50:30.612986    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37fc879e2093a92a12995b998c578c90521c59028b59de2b56258f0336e83760\": container with ID starting with 37fc879e2093a92a12995b998c578c90521c59028b59de2b56258f0336e83760 not found: ID does not exist" containerID="37fc879e2093a92a12995b998c578c90521c59028b59de2b56258f0336e83760"
Jul 30 00:50:30.613029 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.613010    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37fc879e2093a92a12995b998c578c90521c59028b59de2b56258f0336e83760"} err="failed to get container status \"37fc879e2093a92a12995b998c578c90521c59028b59de2b56258f0336e83760\": rpc error: code = NotFound desc = could not find container \"37fc879e2093a92a12995b998c578c90521c59028b59de2b56258f0336e83760\": container with ID starting with 37fc879e2093a92a12995b998c578c90521c59028b59de2b56258f0336e83760 not found: ID does not exist"
Jul 30 00:50:30.613029 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.613028    2579 scope.go:117] "RemoveContainer" containerID="11770a7aa2f1a4c444586d6cf49a0da26512cfb7fc9d4d2318b56bcc03d27a29"
Jul 30 00:50:30.622011 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.621991    2579 scope.go:117] "RemoveContainer" containerID="11770a7aa2f1a4c444586d6cf49a0da26512cfb7fc9d4d2318b56bcc03d27a29"
Jul 30 00:50:30.622304 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:50:30.622275    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11770a7aa2f1a4c444586d6cf49a0da26512cfb7fc9d4d2318b56bcc03d27a29\": container with ID starting with 11770a7aa2f1a4c444586d6cf49a0da26512cfb7fc9d4d2318b56bcc03d27a29 not found: ID does not exist" containerID="11770a7aa2f1a4c444586d6cf49a0da26512cfb7fc9d4d2318b56bcc03d27a29"
Jul 30 00:50:30.622398 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.622312    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11770a7aa2f1a4c444586d6cf49a0da26512cfb7fc9d4d2318b56bcc03d27a29"} err="failed to get container status \"11770a7aa2f1a4c444586d6cf49a0da26512cfb7fc9d4d2318b56bcc03d27a29\": rpc error: code = NotFound desc = could not find container \"11770a7aa2f1a4c444586d6cf49a0da26512cfb7fc9d4d2318b56bcc03d27a29\": container with ID starting with 11770a7aa2f1a4c444586d6cf49a0da26512cfb7fc9d4d2318b56bcc03d27a29 not found: ID does not exist"
Jul 30 00:50:30.622398 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.622334    2579 scope.go:117] "RemoveContainer" containerID="b6fc9a34aeff193715461c9d47dfdcfb03ecf114641368918306f91f849ef50c"
Jul 30 00:50:30.626451 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.626433    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz"]
Jul 30 00:50:30.634153 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.634135    2579 scope.go:117] "RemoveContainer" containerID="b6fc9a34aeff193715461c9d47dfdcfb03ecf114641368918306f91f849ef50c"
Jul 30 00:50:30.634777 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:50:30.634748    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6fc9a34aeff193715461c9d47dfdcfb03ecf114641368918306f91f849ef50c\": container with ID starting with b6fc9a34aeff193715461c9d47dfdcfb03ecf114641368918306f91f849ef50c not found: ID does not exist" containerID="b6fc9a34aeff193715461c9d47dfdcfb03ecf114641368918306f91f849ef50c"
Jul 30 00:50:30.634894 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.634785    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6fc9a34aeff193715461c9d47dfdcfb03ecf114641368918306f91f849ef50c"} err="failed to get container status \"b6fc9a34aeff193715461c9d47dfdcfb03ecf114641368918306f91f849ef50c\": rpc error: code = NotFound desc = could not find container \"b6fc9a34aeff193715461c9d47dfdcfb03ecf114641368918306f91f849ef50c\": container with ID starting with b6fc9a34aeff193715461c9d47dfdcfb03ecf114641368918306f91f849ef50c not found: ID does not exist"
Jul 30 00:50:30.634974 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.634958    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-819bad55-ff7dfdbd8-fp8nz"]
Jul 30 00:50:30.649448 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.649421    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9"]
Jul 30 00:50:30.653374 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.653350    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-819bad55-c85866458-4psp9"]
Jul 30 00:50:30.663498 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.663461    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"]
Jul 30 00:50:30.667179 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:30.667157    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-819bad55-58b8895b67-2w6zs"]
Jul 30 00:50:31.322757 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:31.322722    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c020def-1cb7-4587-929a-da99d5b802f1" path="/var/lib/kubelet/pods/1c020def-1cb7-4587-929a-da99d5b802f1/volumes"
Jul 30 00:50:31.323053 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:31.323040    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf" path="/var/lib/kubelet/pods/4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf/volumes"
Jul 30 00:50:31.323373 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:31.323360    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94b6a154-4679-4c43-9dfe-9919e2bb1439" path="/var/lib/kubelet/pods/94b6a154-4679-4c43-9dfe-9919e2bb1439/volumes"
Jul 30 00:50:31.323706 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:31.323695    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdcd0dec-643d-4a69-a150-6e3ee121eca0" path="/var/lib/kubelet/pods/cdcd0dec-643d-4a69-a150-6e3ee121eca0/volumes"
Jul 30 00:50:59.694223 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.694183    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"]
Jul 30 00:50:59.695575 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.695549    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c020def-1cb7-4587-929a-da99d5b802f1" containerName="payload-pre-processing"
Jul 30 00:50:59.695741 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.695728    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c020def-1cb7-4587-929a-da99d5b802f1" containerName="payload-pre-processing"
Jul 30 00:50:59.695839 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.695829    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdcd0dec-643d-4a69-a150-6e3ee121eca0" containerName="maas-api"
Jul 30 00:50:59.695910 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.695901    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdcd0dec-643d-4a69-a150-6e3ee121eca0" containerName="maas-api"
Jul 30 00:50:59.696014 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.696005    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf" containerName="revoke-keys"
Jul 30 00:50:59.696080 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.696073    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf" containerName="revoke-keys"
Jul 30 00:50:59.696156 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.696147    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b6a154-4679-4c43-9dfe-9919e2bb1439" containerName="payload-processing"
Jul 30 00:50:59.696228 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.696220    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b6a154-4679-4c43-9dfe-9919e2bb1439" containerName="payload-processing"
Jul 30 00:50:59.696507 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.696493    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c95c8e6-e8cb-4fa7-9730-4dbe9079eccf" containerName="revoke-keys"
Jul 30 00:50:59.696616 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.696606    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="94b6a154-4679-4c43-9dfe-9919e2bb1439" containerName="payload-processing"
Jul 30 00:50:59.696695 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.696686    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c020def-1cb7-4587-929a-da99d5b802f1" containerName="payload-pre-processing"
Jul 30 00:50:59.696773 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.696764    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdcd0dec-643d-4a69-a150-6e3ee121eca0" containerName="maas-api"
Jul 30 00:50:59.701208 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.701183    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"
Jul 30 00:50:59.704682 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.704644    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-80d5102b-serving-cert\""
Jul 30 00:50:59.707665 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.707640    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"]
Jul 30 00:50:59.812545 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.812518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846af97e-e45e-4872-8130-aab839ab0184-maas-api-tls\") pod \"maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr\" (UID: \"846af97e-e45e-4872-8130-aab839ab0184\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"
Jul 30 00:50:59.812687 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.812553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846af97e-e45e-4872-8130-aab839ab0184-openshift-service-ca\") pod \"maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr\" (UID: \"846af97e-e45e-4872-8130-aab839ab0184\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"
Jul 30 00:50:59.812687 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.812587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9c48\" (UniqueName: \"kubernetes.io/projected/846af97e-e45e-4872-8130-aab839ab0184-kube-api-access-t9c48\") pod \"maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr\" (UID: \"846af97e-e45e-4872-8130-aab839ab0184\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"
Jul 30 00:50:59.914036 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.913999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846af97e-e45e-4872-8130-aab839ab0184-maas-api-tls\") pod \"maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr\" (UID: \"846af97e-e45e-4872-8130-aab839ab0184\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"
Jul 30 00:50:59.914204 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.914052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846af97e-e45e-4872-8130-aab839ab0184-openshift-service-ca\") pod \"maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr\" (UID: \"846af97e-e45e-4872-8130-aab839ab0184\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"
Jul 30 00:50:59.914204 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.914098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9c48\" (UniqueName: \"kubernetes.io/projected/846af97e-e45e-4872-8130-aab839ab0184-kube-api-access-t9c48\") pod \"maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr\" (UID: \"846af97e-e45e-4872-8130-aab839ab0184\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"
Jul 30 00:50:59.914728 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.914706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846af97e-e45e-4872-8130-aab839ab0184-openshift-service-ca\") pod \"maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr\" (UID: \"846af97e-e45e-4872-8130-aab839ab0184\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"
Jul 30 00:50:59.916259 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.916232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846af97e-e45e-4872-8130-aab839ab0184-maas-api-tls\") pod \"maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr\" (UID: \"846af97e-e45e-4872-8130-aab839ab0184\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"
Jul 30 00:50:59.922138 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:50:59.922114    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9c48\" (UniqueName: \"kubernetes.io/projected/846af97e-e45e-4872-8130-aab839ab0184-kube-api-access-t9c48\") pod \"maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr\" (UID: \"846af97e-e45e-4872-8130-aab839ab0184\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"
Jul 30 00:51:00.014972 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.014898    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"
Jul 30 00:51:00.171640 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.171602    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"]
Jul 30 00:51:00.174049 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:51:00.174015    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846af97e_e45e_4872_8130_aab839ab0184.slice/crio-658af73f4b4392951c4bc97813100f547b6823f4e053f86fcf2ac0d9acef3a1a WatchSource:0}: Error finding container 658af73f4b4392951c4bc97813100f547b6823f4e053f86fcf2ac0d9acef3a1a: Status 404 returned error can't find the container with id 658af73f4b4392951c4bc97813100f547b6823f4e053f86fcf2ac0d9acef3a1a
Jul 30 00:51:00.175437 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.175420    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 00:51:00.393039 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.393009    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w"]
Jul 30 00:51:00.397884 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.397864    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w"
Jul 30 00:51:00.400751 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.400731    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-80d5102b-dockercfg-6zfq8\""
Jul 30 00:51:00.400896 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.400785    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-80d5102b\""
Jul 30 00:51:00.405855 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.405830    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w"]
Jul 30 00:51:00.421106 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.421082    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t"]
Jul 30 00:51:00.424665 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.424644    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t"
Jul 30 00:51:00.432153 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.432127    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t"]
Jul 30 00:51:00.520650 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.520618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr9lb\" (UniqueName: \"kubernetes.io/projected/5a4a4d24-8d83-4a99-af19-37a4eddb2e50-kube-api-access-wr9lb\") pod \"payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w\" (UID: \"5a4a4d24-8d83-4a99-af19-37a4eddb2e50\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w"
Jul 30 00:51:00.520650 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.520653    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4a4d24-8d83-4a99-af19-37a4eddb2e50-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w\" (UID: \"5a4a4d24-8d83-4a99-af19-37a4eddb2e50\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w"
Jul 30 00:51:00.520966 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.520681    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36528c3-d7b1-4c2a-9bff-062f6ad2afdb-plugins-config-volume\") pod \"payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t\" (UID: \"f36528c3-d7b1-4c2a-9bff-062f6ad2afdb\") " pod="openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t"
Jul 30 00:51:00.520966 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.520835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45mcr\" (UniqueName: \"kubernetes.io/projected/f36528c3-d7b1-4c2a-9bff-062f6ad2afdb-kube-api-access-45mcr\") pod \"payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t\" (UID: \"f36528c3-d7b1-4c2a-9bff-062f6ad2afdb\") " pod="openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t"
Jul 30 00:51:00.622062 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.622036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45mcr\" (UniqueName: \"kubernetes.io/projected/f36528c3-d7b1-4c2a-9bff-062f6ad2afdb-kube-api-access-45mcr\") pod \"payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t\" (UID: \"f36528c3-d7b1-4c2a-9bff-062f6ad2afdb\") " pod="openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t"
Jul 30 00:51:00.622163 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.622142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr9lb\" (UniqueName: \"kubernetes.io/projected/5a4a4d24-8d83-4a99-af19-37a4eddb2e50-kube-api-access-wr9lb\") pod \"payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w\" (UID: \"5a4a4d24-8d83-4a99-af19-37a4eddb2e50\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w"
Jul 30 00:51:00.622211 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.622178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4a4d24-8d83-4a99-af19-37a4eddb2e50-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w\" (UID: \"5a4a4d24-8d83-4a99-af19-37a4eddb2e50\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w"
Jul 30 00:51:00.622449 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.622426    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36528c3-d7b1-4c2a-9bff-062f6ad2afdb-plugins-config-volume\") pod \"payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t\" (UID: \"f36528c3-d7b1-4c2a-9bff-062f6ad2afdb\") " pod="openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t"
Jul 30 00:51:00.623094 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.623071    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36528c3-d7b1-4c2a-9bff-062f6ad2afdb-plugins-config-volume\") pod \"payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t\" (UID: \"f36528c3-d7b1-4c2a-9bff-062f6ad2afdb\") " pod="openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t"
Jul 30 00:51:00.623170 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.623118    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4a4d24-8d83-4a99-af19-37a4eddb2e50-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w\" (UID: \"5a4a4d24-8d83-4a99-af19-37a4eddb2e50\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w"
Jul 30 00:51:00.629681 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.629658    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45mcr\" (UniqueName: \"kubernetes.io/projected/f36528c3-d7b1-4c2a-9bff-062f6ad2afdb-kube-api-access-45mcr\") pod \"payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t\" (UID: \"f36528c3-d7b1-4c2a-9bff-062f6ad2afdb\") " pod="openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t"
Jul 30 00:51:00.629793 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.629707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr9lb\" (UniqueName: \"kubernetes.io/projected/5a4a4d24-8d83-4a99-af19-37a4eddb2e50-kube-api-access-wr9lb\") pod \"payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w\" (UID: \"5a4a4d24-8d83-4a99-af19-37a4eddb2e50\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w"
Jul 30 00:51:00.710617 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.710544    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w"
Jul 30 00:51:00.735907 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.735869    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t"
Jul 30 00:51:00.747229 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.747200    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr" event={"ID":"846af97e-e45e-4872-8130-aab839ab0184","Type":"ContainerStarted","Data":"10f7602bb1f4a323c5ba75a9d6fc67da26aec9c71198e447a6085387902f368f"}
Jul 30 00:51:00.747491 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.747440    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr" event={"ID":"846af97e-e45e-4872-8130-aab839ab0184","Type":"ContainerStarted","Data":"658af73f4b4392951c4bc97813100f547b6823f4e053f86fcf2ac0d9acef3a1a"}
Jul 30 00:51:00.764948 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.764819    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr" podStartSLOduration=1.368497378 podStartE2EDuration="1.764796344s" totalImagesPullingTime="396.298966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:51:00.763914464 +0000 UTC m=+1986.061181397" watchObservedRunningTime="2026-07-30 00:51:00.764796344 +0000 UTC m=+1986.062063277"
Jul 30 00:51:00.863372 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.863336    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w"]
Jul 30 00:51:00.865030 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:51:00.864998    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4a4d24_8d83_4a99_af19_37a4eddb2e50.slice/crio-6b68808fc6bb2c244caa8782da85212d88c84aa2a64cc4d73f245c28d6450616 WatchSource:0}: Error finding container 6b68808fc6bb2c244caa8782da85212d88c84aa2a64cc4d73f245c28d6450616: Status 404 returned error can't find the container with id 6b68808fc6bb2c244caa8782da85212d88c84aa2a64cc4d73f245c28d6450616
Jul 30 00:51:00.893932 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:00.893900    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t"]
Jul 30 00:51:00.899372 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:51:00.899347    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36528c3_d7b1_4c2a_9bff_062f6ad2afdb.slice/crio-eb0c6edc107b5423c7d8a69d7f3ea6fc83bd619d5c27281c7576c6461606a5da WatchSource:0}: Error finding container eb0c6edc107b5423c7d8a69d7f3ea6fc83bd619d5c27281c7576c6461606a5da: Status 404 returned error can't find the container with id eb0c6edc107b5423c7d8a69d7f3ea6fc83bd619d5c27281c7576c6461606a5da
Jul 30 00:51:01.753538 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:01.753500    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w" event={"ID":"5a4a4d24-8d83-4a99-af19-37a4eddb2e50","Type":"ContainerStarted","Data":"f001e56522bc68ba6e751e32bcafacdb92b57592a630f623e72e87433c05f718"}
Jul 30 00:51:01.753973 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:01.753566    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w" event={"ID":"5a4a4d24-8d83-4a99-af19-37a4eddb2e50","Type":"ContainerStarted","Data":"6b68808fc6bb2c244caa8782da85212d88c84aa2a64cc4d73f245c28d6450616"}
Jul 30 00:51:01.753973 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:01.753592    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w"
Jul 30 00:51:01.755231 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:01.755207    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t" event={"ID":"f36528c3-d7b1-4c2a-9bff-062f6ad2afdb","Type":"ContainerStarted","Data":"21f15c878738a37368481a0fcff98fbb1f91701b4e4ce42dee99b65f160e74b2"}
Jul 30 00:51:01.755373 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:01.755263    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t" event={"ID":"f36528c3-d7b1-4c2a-9bff-062f6ad2afdb","Type":"ContainerStarted","Data":"eb0c6edc107b5423c7d8a69d7f3ea6fc83bd619d5c27281c7576c6461606a5da"}
Jul 30 00:51:01.755417 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:01.755397    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t"
Jul 30 00:51:01.755462 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:01.755433    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"
Jul 30 00:51:01.770063 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:01.770021    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w" podStartSLOduration=1.7700083 podStartE2EDuration="1.7700083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:51:01.768318586 +0000 UTC m=+1987.065585517" watchObservedRunningTime="2026-07-30 00:51:01.7700083 +0000 UTC m=+1987.067275232"
Jul 30 00:51:01.784811 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:01.784770    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t" podStartSLOduration=1.784759821 podStartE2EDuration="1.784759821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:51:01.783606807 +0000 UTC m=+1987.080873739" watchObservedRunningTime="2026-07-30 00:51:01.784759821 +0000 UTC m=+1987.082026778"
Jul 30 00:51:07.767091 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:07.767060    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"
Jul 30 00:51:12.763880 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:12.763852    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w"
Jul 30 00:51:12.764366 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:12.764039    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t"
Jul 30 00:51:52.396432 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:52.396400    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-80d5102b-rhcb5"]
Jul 30 00:51:52.399953 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:52.399936    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-80d5102b-rhcb5"
Jul 30 00:51:52.402842 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:52.402824    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v8q7j\""
Jul 30 00:51:52.406806 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:52.406771    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-80d5102b-rhcb5"]
Jul 30 00:51:52.469450 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:52.469421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9adc66ca-06f8-46a1-995b-e7fbbe8f04fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-80d5102b-rhcb5\" (UID: \"9adc66ca-06f8-46a1-995b-e7fbbe8f04fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-80d5102b-rhcb5"
Jul 30 00:51:52.570628 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:52.570597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9adc66ca-06f8-46a1-995b-e7fbbe8f04fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-80d5102b-rhcb5\" (UID: \"9adc66ca-06f8-46a1-995b-e7fbbe8f04fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-80d5102b-rhcb5"
Jul 30 00:51:52.571232 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:52.571209    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9adc66ca-06f8-46a1-995b-e7fbbe8f04fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-80d5102b-rhcb5\" (UID: \"9adc66ca-06f8-46a1-995b-e7fbbe8f04fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-80d5102b-rhcb5"
Jul 30 00:51:52.712636 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:52.712564    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-80d5102b-rhcb5"
Jul 30 00:51:52.847002 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:52.846965    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-80d5102b-rhcb5"]
Jul 30 00:51:52.851019 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:51:52.850992    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9adc66ca_06f8_46a1_995b_e7fbbe8f04fb.slice/crio-50e40473ea48739d6ee0d770cf49c3a1efaf62dfb21801615bfa2c7eb6c8ac3e WatchSource:0}: Error finding container 50e40473ea48739d6ee0d770cf49c3a1efaf62dfb21801615bfa2c7eb6c8ac3e: Status 404 returned error can't find the container with id 50e40473ea48739d6ee0d770cf49c3a1efaf62dfb21801615bfa2c7eb6c8ac3e
Jul 30 00:51:53.016180 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:53.016109    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-80d5102b-rhcb5" event={"ID":"9adc66ca-06f8-46a1-995b-e7fbbe8f04fb","Type":"ContainerStarted","Data":"50e40473ea48739d6ee0d770cf49c3a1efaf62dfb21801615bfa2c7eb6c8ac3e"}
Jul 30 00:51:54.022516 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:54.022479    2579 generic.go:356] "Generic (PLEG): container finished" podID="9adc66ca-06f8-46a1-995b-e7fbbe8f04fb" containerID="d030e563636dc06e1227697e6b517fd1c5081402c0676ea0ecd683090986a19c" exitCode=0
Jul 30 00:51:54.022884 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:54.022561    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-80d5102b-rhcb5" event={"ID":"9adc66ca-06f8-46a1-995b-e7fbbe8f04fb","Type":"ContainerDied","Data":"d030e563636dc06e1227697e6b517fd1c5081402c0676ea0ecd683090986a19c"}
Jul 30 00:51:55.161170 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:55.161146    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-80d5102b-rhcb5"
Jul 30 00:51:55.195983 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:55.195950    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9adc66ca-06f8-46a1-995b-e7fbbe8f04fb-maas-api-service-ca\") pod \"9adc66ca-06f8-46a1-995b-e7fbbe8f04fb\" (UID: \"9adc66ca-06f8-46a1-995b-e7fbbe8f04fb\") "
Jul 30 00:51:55.196317 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:55.196296    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9adc66ca-06f8-46a1-995b-e7fbbe8f04fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9adc66ca-06f8-46a1-995b-e7fbbe8f04fb" (UID: "9adc66ca-06f8-46a1-995b-e7fbbe8f04fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:51:55.297702 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:55.297622    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9adc66ca-06f8-46a1-995b-e7fbbe8f04fb-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:51:56.035116 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:56.035089    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-80d5102b-rhcb5"
Jul 30 00:51:56.035313 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:56.035123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-80d5102b-rhcb5" event={"ID":"9adc66ca-06f8-46a1-995b-e7fbbe8f04fb","Type":"ContainerDied","Data":"50e40473ea48739d6ee0d770cf49c3a1efaf62dfb21801615bfa2c7eb6c8ac3e"}
Jul 30 00:51:56.035313 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:51:56.035165    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50e40473ea48739d6ee0d770cf49c3a1efaf62dfb21801615bfa2c7eb6c8ac3e"
Jul 30 00:52:07.908623 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:07.908586    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"]
Jul 30 00:52:07.909008 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:07.908868    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr" podUID="846af97e-e45e-4872-8130-aab839ab0184" containerName="maas-api" containerID="cri-o://10f7602bb1f4a323c5ba75a9d6fc67da26aec9c71198e447a6085387902f368f" gracePeriod=30
Jul 30 00:52:07.965738 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:07.965711    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t"]
Jul 30 00:52:07.965960 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:07.965919    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t" podUID="f36528c3-d7b1-4c2a-9bff-062f6ad2afdb" containerName="payload-processing" containerID="cri-o://21f15c878738a37368481a0fcff98fbb1f91701b4e4ce42dee99b65f160e74b2" gracePeriod=30
Jul 30 00:52:07.970212 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:07.970167    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w"]
Jul 30 00:52:07.971412 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:07.971271    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w" podUID="5a4a4d24-8d83-4a99-af19-37a4eddb2e50" containerName="payload-pre-processing" containerID="cri-o://f001e56522bc68ba6e751e32bcafacdb92b57592a630f623e72e87433c05f718" gracePeriod=30
Jul 30 00:52:08.107507 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.107466    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-80d5102b-rhcb5"]
Jul 30 00:52:08.109943 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.109905    2579 generic.go:356] "Generic (PLEG): container finished" podID="5a4a4d24-8d83-4a99-af19-37a4eddb2e50" containerID="f001e56522bc68ba6e751e32bcafacdb92b57592a630f623e72e87433c05f718" exitCode=0
Jul 30 00:52:08.110093 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.109951    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w" event={"ID":"5a4a4d24-8d83-4a99-af19-37a4eddb2e50","Type":"ContainerDied","Data":"f001e56522bc68ba6e751e32bcafacdb92b57592a630f623e72e87433c05f718"}
Jul 30 00:52:08.110181 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.110158    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-80d5102b-rhcb5"]
Jul 30 00:52:08.114272 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.114062    2579 generic.go:356] "Generic (PLEG): container finished" podID="f36528c3-d7b1-4c2a-9bff-062f6ad2afdb" containerID="21f15c878738a37368481a0fcff98fbb1f91701b4e4ce42dee99b65f160e74b2" exitCode=0
Jul 30 00:52:08.114272 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.114187    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t" event={"ID":"f36528c3-d7b1-4c2a-9bff-062f6ad2afdb","Type":"ContainerDied","Data":"21f15c878738a37368481a0fcff98fbb1f91701b4e4ce42dee99b65f160e74b2"}
Jul 30 00:52:08.116190 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.116156    2579 generic.go:356] "Generic (PLEG): container finished" podID="846af97e-e45e-4872-8130-aab839ab0184" containerID="10f7602bb1f4a323c5ba75a9d6fc67da26aec9c71198e447a6085387902f368f" exitCode=0
Jul 30 00:52:08.116342 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.116199    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr" event={"ID":"846af97e-e45e-4872-8130-aab839ab0184","Type":"ContainerDied","Data":"10f7602bb1f4a323c5ba75a9d6fc67da26aec9c71198e447a6085387902f368f"}
Jul 30 00:52:08.245440 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.245419    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t"
Jul 30 00:52:08.248163 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.248144    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"
Jul 30 00:52:08.266189 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.266157    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w"
Jul 30 00:52:08.416546 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.416515    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr9lb\" (UniqueName: \"kubernetes.io/projected/5a4a4d24-8d83-4a99-af19-37a4eddb2e50-kube-api-access-wr9lb\") pod \"5a4a4d24-8d83-4a99-af19-37a4eddb2e50\" (UID: \"5a4a4d24-8d83-4a99-af19-37a4eddb2e50\") "
Jul 30 00:52:08.416723 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.416571    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36528c3-d7b1-4c2a-9bff-062f6ad2afdb-plugins-config-volume\") pod \"f36528c3-d7b1-4c2a-9bff-062f6ad2afdb\" (UID: \"f36528c3-d7b1-4c2a-9bff-062f6ad2afdb\") "
Jul 30 00:52:08.416723 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.416596    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9c48\" (UniqueName: \"kubernetes.io/projected/846af97e-e45e-4872-8130-aab839ab0184-kube-api-access-t9c48\") pod \"846af97e-e45e-4872-8130-aab839ab0184\" (UID: \"846af97e-e45e-4872-8130-aab839ab0184\") "
Jul 30 00:52:08.416723 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.416630    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846af97e-e45e-4872-8130-aab839ab0184-maas-api-tls\") pod \"846af97e-e45e-4872-8130-aab839ab0184\" (UID: \"846af97e-e45e-4872-8130-aab839ab0184\") "
Jul 30 00:52:08.416723 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.416646    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45mcr\" (UniqueName: \"kubernetes.io/projected/f36528c3-d7b1-4c2a-9bff-062f6ad2afdb-kube-api-access-45mcr\") pod \"f36528c3-d7b1-4c2a-9bff-062f6ad2afdb\" (UID: \"f36528c3-d7b1-4c2a-9bff-062f6ad2afdb\") "
Jul 30 00:52:08.416723 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.416692    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4a4d24-8d83-4a99-af19-37a4eddb2e50-plugins-config-volume\") pod \"5a4a4d24-8d83-4a99-af19-37a4eddb2e50\" (UID: \"5a4a4d24-8d83-4a99-af19-37a4eddb2e50\") "
Jul 30 00:52:08.417000 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.416730    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846af97e-e45e-4872-8130-aab839ab0184-openshift-service-ca\") pod \"846af97e-e45e-4872-8130-aab839ab0184\" (UID: \"846af97e-e45e-4872-8130-aab839ab0184\") "
Jul 30 00:52:08.417192 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.417165    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846af97e-e45e-4872-8130-aab839ab0184-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "846af97e-e45e-4872-8130-aab839ab0184" (UID: "846af97e-e45e-4872-8130-aab839ab0184"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:52:08.417634 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.417564    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36528c3-d7b1-4c2a-9bff-062f6ad2afdb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f36528c3-d7b1-4c2a-9bff-062f6ad2afdb" (UID: "f36528c3-d7b1-4c2a-9bff-062f6ad2afdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:52:08.417800 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.417765    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4a4d24-8d83-4a99-af19-37a4eddb2e50-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a4a4d24-8d83-4a99-af19-37a4eddb2e50" (UID: "5a4a4d24-8d83-4a99-af19-37a4eddb2e50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:52:08.418965 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.418945    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846af97e-e45e-4872-8130-aab839ab0184-kube-api-access-t9c48" (OuterVolumeSpecName: "kube-api-access-t9c48") pod "846af97e-e45e-4872-8130-aab839ab0184" (UID: "846af97e-e45e-4872-8130-aab839ab0184"). InnerVolumeSpecName "kube-api-access-t9c48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:52:08.419027 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.418996    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36528c3-d7b1-4c2a-9bff-062f6ad2afdb-kube-api-access-45mcr" (OuterVolumeSpecName: "kube-api-access-45mcr") pod "f36528c3-d7b1-4c2a-9bff-062f6ad2afdb" (UID: "f36528c3-d7b1-4c2a-9bff-062f6ad2afdb"). InnerVolumeSpecName "kube-api-access-45mcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:52:08.419149 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.419131    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4a4d24-8d83-4a99-af19-37a4eddb2e50-kube-api-access-wr9lb" (OuterVolumeSpecName: "kube-api-access-wr9lb") pod "5a4a4d24-8d83-4a99-af19-37a4eddb2e50" (UID: "5a4a4d24-8d83-4a99-af19-37a4eddb2e50"). InnerVolumeSpecName "kube-api-access-wr9lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:52:08.419149 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.419138    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/846af97e-e45e-4872-8130-aab839ab0184-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "846af97e-e45e-4872-8130-aab839ab0184" (UID: "846af97e-e45e-4872-8130-aab839ab0184"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:52:08.517753 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.517694    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4a4d24-8d83-4a99-af19-37a4eddb2e50-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:52:08.517753 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.517718    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846af97e-e45e-4872-8130-aab839ab0184-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:52:08.517753 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.517727    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr9lb\" (UniqueName: \"kubernetes.io/projected/5a4a4d24-8d83-4a99-af19-37a4eddb2e50-kube-api-access-wr9lb\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:52:08.517753 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.517735    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36528c3-d7b1-4c2a-9bff-062f6ad2afdb-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:52:08.517753 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.517745    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9c48\" (UniqueName: \"kubernetes.io/projected/846af97e-e45e-4872-8130-aab839ab0184-kube-api-access-t9c48\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:52:08.517753 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.517753    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846af97e-e45e-4872-8130-aab839ab0184-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:52:08.518010 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:08.517763    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45mcr\" (UniqueName: \"kubernetes.io/projected/f36528c3-d7b1-4c2a-9bff-062f6ad2afdb-kube-api-access-45mcr\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:52:09.122643 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:09.122608    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w"
Jul 30 00:52:09.123082 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:09.122652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w" event={"ID":"5a4a4d24-8d83-4a99-af19-37a4eddb2e50","Type":"ContainerDied","Data":"6b68808fc6bb2c244caa8782da85212d88c84aa2a64cc4d73f245c28d6450616"}
Jul 30 00:52:09.123082 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:09.122718    2579 scope.go:117] "RemoveContainer" containerID="f001e56522bc68ba6e751e32bcafacdb92b57592a630f623e72e87433c05f718"
Jul 30 00:52:09.124100 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:09.124079    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t"
Jul 30 00:52:09.124189 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:09.124118    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t" event={"ID":"f36528c3-d7b1-4c2a-9bff-062f6ad2afdb","Type":"ContainerDied","Data":"eb0c6edc107b5423c7d8a69d7f3ea6fc83bd619d5c27281c7576c6461606a5da"}
Jul 30 00:52:09.125564 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:09.125535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr" event={"ID":"846af97e-e45e-4872-8130-aab839ab0184","Type":"ContainerDied","Data":"658af73f4b4392951c4bc97813100f547b6823f4e053f86fcf2ac0d9acef3a1a"}
Jul 30 00:52:09.125564 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:09.125541    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"
Jul 30 00:52:09.132780 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:09.132762    2579 scope.go:117] "RemoveContainer" containerID="21f15c878738a37368481a0fcff98fbb1f91701b4e4ce42dee99b65f160e74b2"
Jul 30 00:52:09.143781 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:09.143758    2579 scope.go:117] "RemoveContainer" containerID="10f7602bb1f4a323c5ba75a9d6fc67da26aec9c71198e447a6085387902f368f"
Jul 30 00:52:09.150808 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:09.150787    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w"]
Jul 30 00:52:09.156326 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:09.156306    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-80d5102b-5d8f65976d-j677w"]
Jul 30 00:52:09.169686 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:09.169663    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t"]
Jul 30 00:52:09.175538 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:09.175487    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-80d5102b-5cdfdc45d6-fzj9t"]
Jul 30 00:52:09.185699 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:09.185679    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"]
Jul 30 00:52:09.189363 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:09.189343    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-80d5102b-7cbb4dd87d-7dbtr"]
Jul 30 00:52:09.322613 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:09.322583    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a4a4d24-8d83-4a99-af19-37a4eddb2e50" path="/var/lib/kubelet/pods/5a4a4d24-8d83-4a99-af19-37a4eddb2e50/volumes"
Jul 30 00:52:09.322974 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:09.322961    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="846af97e-e45e-4872-8130-aab839ab0184" path="/var/lib/kubelet/pods/846af97e-e45e-4872-8130-aab839ab0184/volumes"
Jul 30 00:52:09.323260 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:09.323234    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9adc66ca-06f8-46a1-995b-e7fbbe8f04fb" path="/var/lib/kubelet/pods/9adc66ca-06f8-46a1-995b-e7fbbe8f04fb/volumes"
Jul 30 00:52:09.323575 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:09.323562    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f36528c3-d7b1-4c2a-9bff-062f6ad2afdb" path="/var/lib/kubelet/pods/f36528c3-d7b1-4c2a-9bff-062f6ad2afdb/volumes"
Jul 30 00:52:55.317054 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:55.317019    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:52:55.319083 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:55.319052    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:52:55.447000 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:55.446967    2579 scope.go:117] "RemoveContainer" containerID="acd5f4658c1a68a93397a8f276cc419c46b26cb4ce2ae105675d9fb576e4a152"
Jul 30 00:52:55.455600 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:52:55.455581    2579 scope.go:117] "RemoveContainer" containerID="fc3612f855e66a296f589d2b864cf4549b7bfede515427678cfa65a612af9d8f"
Jul 30 00:53:55.491375 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:53:55.491338    2579 scope.go:117] "RemoveContainer" containerID="c72ece906a17a91c8f80edd8e30fc631e9b8b6e0c9f90b1641067f5f24630e92"
Jul 30 00:53:55.499431 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:53:55.499415    2579 scope.go:117] "RemoveContainer" containerID="28afe07d807a8a28bdf2098ec8be620ef80f3fba9de451a1f265d3256f506ea2"
Jul 30 00:54:03.698001 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.697970    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"]
Jul 30 00:54:03.698560 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.698540    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f36528c3-d7b1-4c2a-9bff-062f6ad2afdb" containerName="payload-processing"
Jul 30 00:54:03.698649 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.698563    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f36528c3-d7b1-4c2a-9bff-062f6ad2afdb" containerName="payload-processing"
Jul 30 00:54:03.698649 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.698582    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4a4d24-8d83-4a99-af19-37a4eddb2e50" containerName="payload-pre-processing"
Jul 30 00:54:03.698649 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.698591    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4a4d24-8d83-4a99-af19-37a4eddb2e50" containerName="payload-pre-processing"
Jul 30 00:54:03.698649 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.698606    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="846af97e-e45e-4872-8130-aab839ab0184" containerName="maas-api"
Jul 30 00:54:03.698649 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.698615    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="846af97e-e45e-4872-8130-aab839ab0184" containerName="maas-api"
Jul 30 00:54:03.698649 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.698642    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9adc66ca-06f8-46a1-995b-e7fbbe8f04fb" containerName="revoke-keys"
Jul 30 00:54:03.698649 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.698650    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9adc66ca-06f8-46a1-995b-e7fbbe8f04fb" containerName="revoke-keys"
Jul 30 00:54:03.698993 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.698759    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="846af97e-e45e-4872-8130-aab839ab0184" containerName="maas-api"
Jul 30 00:54:03.698993 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.698776    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9adc66ca-06f8-46a1-995b-e7fbbe8f04fb" containerName="revoke-keys"
Jul 30 00:54:03.698993 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.698788    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f36528c3-d7b1-4c2a-9bff-062f6ad2afdb" containerName="payload-processing"
Jul 30 00:54:03.698993 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.698799    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a4a4d24-8d83-4a99-af19-37a4eddb2e50" containerName="payload-pre-processing"
Jul 30 00:54:03.701968 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.701947    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"
Jul 30 00:54:03.704505 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.704480    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2239f6-serving-cert\""
Jul 30 00:54:03.710159 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.709918    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"]
Jul 30 00:54:03.714308 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.714286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc\" (UID: \"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"
Jul 30 00:54:03.714408 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.714322    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-maas-api-tls\") pod \"maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc\" (UID: \"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"
Jul 30 00:54:03.714408 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.714341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxghx\" (UniqueName: \"kubernetes.io/projected/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-kube-api-access-cxghx\") pod \"maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc\" (UID: \"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"
Jul 30 00:54:03.815031 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.814999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc\" (UID: \"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"
Jul 30 00:54:03.815171 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.815040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-maas-api-tls\") pod \"maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc\" (UID: \"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"
Jul 30 00:54:03.815171 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.815067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxghx\" (UniqueName: \"kubernetes.io/projected/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-kube-api-access-cxghx\") pod \"maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc\" (UID: \"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"
Jul 30 00:54:03.815171 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:54:03.815161    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-serving-cert: secret "maas-api-e2e-inf-a-2239f6-serving-cert" not found
Jul 30 00:54:03.815307 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:54:03.815228    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-maas-api-tls podName:d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a nodeName:}" failed. No retries permitted until 2026-07-30 00:54:04.315204767 +0000 UTC m=+2169.612471680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-maas-api-tls") pod "maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc" (UID: "d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a") : secret "maas-api-e2e-inf-a-2239f6-serving-cert" not found
Jul 30 00:54:03.815651 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.815633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc\" (UID: \"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"
Jul 30 00:54:03.823450 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:03.823433    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxghx\" (UniqueName: \"kubernetes.io/projected/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-kube-api-access-cxghx\") pod \"maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc\" (UID: \"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"
Jul 30 00:54:04.260625 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.260593    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2"]
Jul 30 00:54:04.264251 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.264224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2"
Jul 30 00:54:04.267637 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.267617    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2239f6-dockercfg-zmrvn\""
Jul 30 00:54:04.267754 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.267649    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2239f6\""
Jul 30 00:54:04.274385 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.274365    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2"]
Jul 30 00:54:04.295541 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.295510    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw"]
Jul 30 00:54:04.298931 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.298912    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw"
Jul 30 00:54:04.306751 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.306730    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw"]
Jul 30 00:54:04.319169 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.319143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ckjs\" (UniqueName: \"kubernetes.io/projected/13afd1e6-cd05-489f-acd3-f4f907acbbb5-kube-api-access-9ckjs\") pod \"payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw\" (UID: \"13afd1e6-cd05-489f-acd3-f4f907acbbb5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw"
Jul 30 00:54:04.319316 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.319225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9208c47e-7c14-4f3b-9076-7fe33aec43d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2\" (UID: \"9208c47e-7c14-4f3b-9076-7fe33aec43d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2"
Jul 30 00:54:04.319316 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.319272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13afd1e6-cd05-489f-acd3-f4f907acbbb5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw\" (UID: \"13afd1e6-cd05-489f-acd3-f4f907acbbb5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw"
Jul 30 00:54:04.319408 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.319376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-maas-api-tls\") pod \"maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc\" (UID: \"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"
Jul 30 00:54:04.319487 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.319469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7gqj\" (UniqueName: \"kubernetes.io/projected/9208c47e-7c14-4f3b-9076-7fe33aec43d2-kube-api-access-r7gqj\") pod \"payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2\" (UID: \"9208c47e-7c14-4f3b-9076-7fe33aec43d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2"
Jul 30 00:54:04.322081 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.322051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-maas-api-tls\") pod \"maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc\" (UID: \"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"
Jul 30 00:54:04.420339 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.420305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ckjs\" (UniqueName: \"kubernetes.io/projected/13afd1e6-cd05-489f-acd3-f4f907acbbb5-kube-api-access-9ckjs\") pod \"payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw\" (UID: \"13afd1e6-cd05-489f-acd3-f4f907acbbb5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw"
Jul 30 00:54:04.420492 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.420362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9208c47e-7c14-4f3b-9076-7fe33aec43d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2\" (UID: \"9208c47e-7c14-4f3b-9076-7fe33aec43d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2"
Jul 30 00:54:04.420565 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.420544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13afd1e6-cd05-489f-acd3-f4f907acbbb5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw\" (UID: \"13afd1e6-cd05-489f-acd3-f4f907acbbb5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw"
Jul 30 00:54:04.420694 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.420677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7gqj\" (UniqueName: \"kubernetes.io/projected/9208c47e-7c14-4f3b-9076-7fe33aec43d2-kube-api-access-r7gqj\") pod \"payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2\" (UID: \"9208c47e-7c14-4f3b-9076-7fe33aec43d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2"
Jul 30 00:54:04.421040 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.421020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9208c47e-7c14-4f3b-9076-7fe33aec43d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2\" (UID: \"9208c47e-7c14-4f3b-9076-7fe33aec43d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2"
Jul 30 00:54:04.421211 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.421193    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13afd1e6-cd05-489f-acd3-f4f907acbbb5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw\" (UID: \"13afd1e6-cd05-489f-acd3-f4f907acbbb5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw"
Jul 30 00:54:04.428370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.428345    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ckjs\" (UniqueName: \"kubernetes.io/projected/13afd1e6-cd05-489f-acd3-f4f907acbbb5-kube-api-access-9ckjs\") pod \"payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw\" (UID: \"13afd1e6-cd05-489f-acd3-f4f907acbbb5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw"
Jul 30 00:54:04.428370 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.428358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7gqj\" (UniqueName: \"kubernetes.io/projected/9208c47e-7c14-4f3b-9076-7fe33aec43d2-kube-api-access-r7gqj\") pod \"payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2\" (UID: \"9208c47e-7c14-4f3b-9076-7fe33aec43d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2"
Jul 30 00:54:04.575433 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.575362    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2"
Jul 30 00:54:04.609868 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.609833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw"
Jul 30 00:54:04.614503 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.614483    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"
Jul 30 00:54:04.736120 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.735964    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2"]
Jul 30 00:54:04.739475 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:54:04.739444    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9208c47e_7c14_4f3b_9076_7fe33aec43d2.slice/crio-e68cc96bd0bdc26d493698379c7f3aafedd370cecf50ad89bf71fd5e2e0b631b WatchSource:0}: Error finding container e68cc96bd0bdc26d493698379c7f3aafedd370cecf50ad89bf71fd5e2e0b631b: Status 404 returned error can't find the container with id e68cc96bd0bdc26d493698379c7f3aafedd370cecf50ad89bf71fd5e2e0b631b
Jul 30 00:54:04.991818 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:04.991794    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw"]
Jul 30 00:54:04.994625 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:54:04.994594    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13afd1e6_cd05_489f_acd3_f4f907acbbb5.slice/crio-0977668348fe2fc691842d519fa6b5e56abff07f6dfa789d9b9b1f24f0934e44 WatchSource:0}: Error finding container 0977668348fe2fc691842d519fa6b5e56abff07f6dfa789d9b9b1f24f0934e44: Status 404 returned error can't find the container with id 0977668348fe2fc691842d519fa6b5e56abff07f6dfa789d9b9b1f24f0934e44
Jul 30 00:54:05.015188 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:05.015157    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"]
Jul 30 00:54:05.017765 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:54:05.017734    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b93ceb_7ca2_4e68_b331_2c2bbc66e51a.slice/crio-5fb301290808e544dc1019ca425a46043c2d9d4d626e344fd1f392ac27dfbba4 WatchSource:0}: Error finding container 5fb301290808e544dc1019ca425a46043c2d9d4d626e344fd1f392ac27dfbba4: Status 404 returned error can't find the container with id 5fb301290808e544dc1019ca425a46043c2d9d4d626e344fd1f392ac27dfbba4
Jul 30 00:54:05.712819 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:05.712737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2" event={"ID":"9208c47e-7c14-4f3b-9076-7fe33aec43d2","Type":"ContainerStarted","Data":"7c20265d69736a17f480e63cd496fe1283c66dcd68e6952d30922393a34aa1d5"}
Jul 30 00:54:05.712819 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:05.712792    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2" event={"ID":"9208c47e-7c14-4f3b-9076-7fe33aec43d2","Type":"ContainerStarted","Data":"e68cc96bd0bdc26d493698379c7f3aafedd370cecf50ad89bf71fd5e2e0b631b"}
Jul 30 00:54:05.713050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:05.712834    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2"
Jul 30 00:54:05.714439 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:05.714411    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc" event={"ID":"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a","Type":"ContainerStarted","Data":"81811cca3f080682c0c68d60dca8fc87df419ee608e89f37a7e6f0b7cca9c67d"}
Jul 30 00:54:05.714575 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:05.714454    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc" event={"ID":"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a","Type":"ContainerStarted","Data":"5fb301290808e544dc1019ca425a46043c2d9d4d626e344fd1f392ac27dfbba4"}
Jul 30 00:54:05.714575 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:05.714484    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"
Jul 30 00:54:05.715804 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:05.715780    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw" event={"ID":"13afd1e6-cd05-489f-acd3-f4f907acbbb5","Type":"ContainerStarted","Data":"b2d88a9960930ddd26acd6c1693be7c54d1718e8a85d01390ff67339172ff8eb"}
Jul 30 00:54:05.715913 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:05.715821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw" event={"ID":"13afd1e6-cd05-489f-acd3-f4f907acbbb5","Type":"ContainerStarted","Data":"0977668348fe2fc691842d519fa6b5e56abff07f6dfa789d9b9b1f24f0934e44"}
Jul 30 00:54:05.715913 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:05.715855    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw"
Jul 30 00:54:05.728928 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:05.728883    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2" podStartSLOduration=1.728870175 podStartE2EDuration="1.728870175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:54:05.728562251 +0000 UTC m=+2171.025829183" watchObservedRunningTime="2026-07-30 00:54:05.728870175 +0000 UTC m=+2171.026137107"
Jul 30 00:54:05.743956 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:05.743906    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw" podStartSLOduration=1.743894574 podStartE2EDuration="1.743894574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:54:05.742223284 +0000 UTC m=+2171.039490230" watchObservedRunningTime="2026-07-30 00:54:05.743894574 +0000 UTC m=+2171.041161505"
Jul 30 00:54:05.759802 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:05.759765    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc" podStartSLOduration=2.317556832 podStartE2EDuration="2.759751231s" totalImagesPullingTime="442.194399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:54:05.757264171 +0000 UTC m=+2171.054531099" watchObservedRunningTime="2026-07-30 00:54:05.759751231 +0000 UTC m=+2171.057018163"
Jul 30 00:54:11.725919 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:11.725886    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"
Jul 30 00:54:16.723179 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:16.723148    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw"
Jul 30 00:54:16.723619 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:16.723233    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2"
Jul 30 00:54:31.887233 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:31.887200    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"]
Jul 30 00:54:31.890702 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:31.890677    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"
Jul 30 00:54:31.893523 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:31.893493    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6898ea-serving-cert\""
Jul 30 00:54:31.902026 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:31.902002    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"]
Jul 30 00:54:31.981651 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:31.981621    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n452x\" (UniqueName: \"kubernetes.io/projected/299ca293-6666-4d2a-bddd-624afdf6be85-kube-api-access-n452x\") pod \"maas-api-e2e-inf-b-6898ea-647995bc98-c88qx\" (UID: \"299ca293-6666-4d2a-bddd-624afdf6be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"
Jul 30 00:54:31.981803 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:31.981693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/299ca293-6666-4d2a-bddd-624afdf6be85-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6898ea-647995bc98-c88qx\" (UID: \"299ca293-6666-4d2a-bddd-624afdf6be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"
Jul 30 00:54:31.981803 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:31.981724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/299ca293-6666-4d2a-bddd-624afdf6be85-maas-api-tls\") pod \"maas-api-e2e-inf-b-6898ea-647995bc98-c88qx\" (UID: \"299ca293-6666-4d2a-bddd-624afdf6be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"
Jul 30 00:54:32.082725 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.082682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/299ca293-6666-4d2a-bddd-624afdf6be85-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6898ea-647995bc98-c88qx\" (UID: \"299ca293-6666-4d2a-bddd-624afdf6be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"
Jul 30 00:54:32.082725 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.082732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/299ca293-6666-4d2a-bddd-624afdf6be85-maas-api-tls\") pod \"maas-api-e2e-inf-b-6898ea-647995bc98-c88qx\" (UID: \"299ca293-6666-4d2a-bddd-624afdf6be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"
Jul 30 00:54:32.082999 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.082780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n452x\" (UniqueName: \"kubernetes.io/projected/299ca293-6666-4d2a-bddd-624afdf6be85-kube-api-access-n452x\") pod \"maas-api-e2e-inf-b-6898ea-647995bc98-c88qx\" (UID: \"299ca293-6666-4d2a-bddd-624afdf6be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"
Jul 30 00:54:32.082999 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:54:32.082899    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-serving-cert: secret "maas-api-e2e-inf-b-6898ea-serving-cert" not found
Jul 30 00:54:32.082999 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:54:32.082978    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/299ca293-6666-4d2a-bddd-624afdf6be85-maas-api-tls podName:299ca293-6666-4d2a-bddd-624afdf6be85 nodeName:}" failed. No retries permitted until 2026-07-30 00:54:32.582959329 +0000 UTC m=+2197.880226239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/299ca293-6666-4d2a-bddd-624afdf6be85-maas-api-tls") pod "maas-api-e2e-inf-b-6898ea-647995bc98-c88qx" (UID: "299ca293-6666-4d2a-bddd-624afdf6be85") : secret "maas-api-e2e-inf-b-6898ea-serving-cert" not found
Jul 30 00:54:32.083483 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.083457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/299ca293-6666-4d2a-bddd-624afdf6be85-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6898ea-647995bc98-c88qx\" (UID: \"299ca293-6666-4d2a-bddd-624afdf6be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"
Jul 30 00:54:32.093969 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.093936    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n452x\" (UniqueName: \"kubernetes.io/projected/299ca293-6666-4d2a-bddd-624afdf6be85-kube-api-access-n452x\") pod \"maas-api-e2e-inf-b-6898ea-647995bc98-c88qx\" (UID: \"299ca293-6666-4d2a-bddd-624afdf6be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"
Jul 30 00:54:32.448121 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.448093    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx"]
Jul 30 00:54:32.451733 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.451713    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx"
Jul 30 00:54:32.455472 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.455451    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6898ea\""
Jul 30 00:54:32.455607 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.455535    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6898ea-dockercfg-b8qs7\""
Jul 30 00:54:32.462445 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.462423    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx"]
Jul 30 00:54:32.485070 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.485044    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8"]
Jul 30 00:54:32.488438 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.488421    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8"
Jul 30 00:54:32.497398 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.497375    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8"]
Jul 30 00:54:32.587685 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.587651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29fe2a57-d801-4c9b-bd73-caaf8013f94d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8\" (UID: \"29fe2a57-d801-4c9b-bd73-caaf8013f94d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8"
Jul 30 00:54:32.587841 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.587702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sgr9\" (UniqueName: \"kubernetes.io/projected/29fe2a57-d801-4c9b-bd73-caaf8013f94d-kube-api-access-4sgr9\") pod \"payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8\" (UID: \"29fe2a57-d801-4c9b-bd73-caaf8013f94d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8"
Jul 30 00:54:32.587841 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.587808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mxck\" (UniqueName: \"kubernetes.io/projected/79857008-e5dc-488b-809a-fb547fc88bdc-kube-api-access-5mxck\") pod \"payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx\" (UID: \"79857008-e5dc-488b-809a-fb547fc88bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx"
Jul 30 00:54:32.587920 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.587849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79857008-e5dc-488b-809a-fb547fc88bdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx\" (UID: \"79857008-e5dc-488b-809a-fb547fc88bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx"
Jul 30 00:54:32.587972 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.587953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/299ca293-6666-4d2a-bddd-624afdf6be85-maas-api-tls\") pod \"maas-api-e2e-inf-b-6898ea-647995bc98-c88qx\" (UID: \"299ca293-6666-4d2a-bddd-624afdf6be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"
Jul 30 00:54:32.590382 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.590362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/299ca293-6666-4d2a-bddd-624afdf6be85-maas-api-tls\") pod \"maas-api-e2e-inf-b-6898ea-647995bc98-c88qx\" (UID: \"299ca293-6666-4d2a-bddd-624afdf6be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"
Jul 30 00:54:32.688695 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.688662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29fe2a57-d801-4c9b-bd73-caaf8013f94d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8\" (UID: \"29fe2a57-d801-4c9b-bd73-caaf8013f94d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8"
Jul 30 00:54:32.688695 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.688697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgr9\" (UniqueName: \"kubernetes.io/projected/29fe2a57-d801-4c9b-bd73-caaf8013f94d-kube-api-access-4sgr9\") pod \"payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8\" (UID: \"29fe2a57-d801-4c9b-bd73-caaf8013f94d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8"
Jul 30 00:54:32.688914 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.688743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mxck\" (UniqueName: \"kubernetes.io/projected/79857008-e5dc-488b-809a-fb547fc88bdc-kube-api-access-5mxck\") pod \"payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx\" (UID: \"79857008-e5dc-488b-809a-fb547fc88bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx"
Jul 30 00:54:32.688914 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.688763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79857008-e5dc-488b-809a-fb547fc88bdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx\" (UID: \"79857008-e5dc-488b-809a-fb547fc88bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx"
Jul 30 00:54:32.689486 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.689467    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79857008-e5dc-488b-809a-fb547fc88bdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx\" (UID: \"79857008-e5dc-488b-809a-fb547fc88bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx"
Jul 30 00:54:32.689574 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.689534    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29fe2a57-d801-4c9b-bd73-caaf8013f94d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8\" (UID: \"29fe2a57-d801-4c9b-bd73-caaf8013f94d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8"
Jul 30 00:54:32.696969 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.696949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mxck\" (UniqueName: \"kubernetes.io/projected/79857008-e5dc-488b-809a-fb547fc88bdc-kube-api-access-5mxck\") pod \"payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx\" (UID: \"79857008-e5dc-488b-809a-fb547fc88bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx"
Jul 30 00:54:32.697058 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.696951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sgr9\" (UniqueName: \"kubernetes.io/projected/29fe2a57-d801-4c9b-bd73-caaf8013f94d-kube-api-access-4sgr9\") pod \"payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8\" (UID: \"29fe2a57-d801-4c9b-bd73-caaf8013f94d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8"
Jul 30 00:54:32.764122 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.764052    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx"
Jul 30 00:54:32.800215 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.800183    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8"
Jul 30 00:54:32.802381 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.802352    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"
Jul 30 00:54:32.932495 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.932460    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx"]
Jul 30 00:54:32.934792 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:54:32.934760    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79857008_e5dc_488b_809a_fb547fc88bdc.slice/crio-c427ca6949924a5cacf2ff510bf24bce4bf35039df78248dbf3d1a9ffa7e758a WatchSource:0}: Error finding container c427ca6949924a5cacf2ff510bf24bce4bf35039df78248dbf3d1a9ffa7e758a: Status 404 returned error can't find the container with id c427ca6949924a5cacf2ff510bf24bce4bf35039df78248dbf3d1a9ffa7e758a
Jul 30 00:54:32.983183 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:32.983150    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"]
Jul 30 00:54:32.987832 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:54:32.987805    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299ca293_6666_4d2a_bddd_624afdf6be85.slice/crio-2cc38a49d1c522ef8cf0bb3bbcb47d6a4eab561893ad984fd7aa30a6ada9074a WatchSource:0}: Error finding container 2cc38a49d1c522ef8cf0bb3bbcb47d6a4eab561893ad984fd7aa30a6ada9074a: Status 404 returned error can't find the container with id 2cc38a49d1c522ef8cf0bb3bbcb47d6a4eab561893ad984fd7aa30a6ada9074a
Jul 30 00:54:33.213050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:33.213020    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8"]
Jul 30 00:54:33.214590 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:54:33.214556    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29fe2a57_d801_4c9b_bd73_caaf8013f94d.slice/crio-a7d139b1dfdc896355f32b42a65e0731844cf4d41f2e939efe8b60d295792b1f WatchSource:0}: Error finding container a7d139b1dfdc896355f32b42a65e0731844cf4d41f2e939efe8b60d295792b1f: Status 404 returned error can't find the container with id a7d139b1dfdc896355f32b42a65e0731844cf4d41f2e939efe8b60d295792b1f
Jul 30 00:54:33.869185 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:33.869145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx" event={"ID":"299ca293-6666-4d2a-bddd-624afdf6be85","Type":"ContainerStarted","Data":"2cc38a49d1c522ef8cf0bb3bbcb47d6a4eab561893ad984fd7aa30a6ada9074a"}
Jul 30 00:54:33.870690 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:33.870655    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8" event={"ID":"29fe2a57-d801-4c9b-bd73-caaf8013f94d","Type":"ContainerStarted","Data":"190d9ec577b3f0f3780c7149135aee860b37704fcc2ea72b12253c3722f612ea"}
Jul 30 00:54:33.870812 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:33.870713    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8" event={"ID":"29fe2a57-d801-4c9b-bd73-caaf8013f94d","Type":"ContainerStarted","Data":"a7d139b1dfdc896355f32b42a65e0731844cf4d41f2e939efe8b60d295792b1f"}
Jul 30 00:54:33.870812 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:33.870755    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8"
Jul 30 00:54:33.872155 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:33.872132    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx" event={"ID":"79857008-e5dc-488b-809a-fb547fc88bdc","Type":"ContainerStarted","Data":"5dbc7be15924baaa8e6b9c438f9ba040538243cc72c9e7db15fb167eeabb6422"}
Jul 30 00:54:33.872262 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:33.872177    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx" event={"ID":"79857008-e5dc-488b-809a-fb547fc88bdc","Type":"ContainerStarted","Data":"c427ca6949924a5cacf2ff510bf24bce4bf35039df78248dbf3d1a9ffa7e758a"}
Jul 30 00:54:33.872262 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:33.872219    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx"
Jul 30 00:54:33.887814 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:33.887771    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8" podStartSLOduration=1.88775806 podStartE2EDuration="1.88775806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:54:33.887227201 +0000 UTC m=+2199.184494133" watchObservedRunningTime="2026-07-30 00:54:33.88775806 +0000 UTC m=+2199.185024970"
Jul 30 00:54:33.906361 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:33.906319    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx" podStartSLOduration=1.906306468 podStartE2EDuration="1.906306468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:54:33.90417722 +0000 UTC m=+2199.201444146" watchObservedRunningTime="2026-07-30 00:54:33.906306468 +0000 UTC m=+2199.203573399"
Jul 30 00:54:34.879159 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:34.879124    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx" event={"ID":"299ca293-6666-4d2a-bddd-624afdf6be85","Type":"ContainerStarted","Data":"c8fd637660bcfa812af7ea2db37ded4f201fb275c8988369a7797c3fb01d2e53"}
Jul 30 00:54:34.879601 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:34.879270    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"
Jul 30 00:54:34.897949 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:34.897896    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx" podStartSLOduration=2.973681784 podStartE2EDuration="3.897881983s" totalImagesPullingTime="924.200199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:54:34.895607845 +0000 UTC m=+2200.192874788" watchObservedRunningTime="2026-07-30 00:54:34.897881983 +0000 UTC m=+2200.195149250"
Jul 30 00:54:40.889986 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:40.889957    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"
Jul 30 00:54:44.881945 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:44.881915    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8"
Jul 30 00:54:44.882426 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:54:44.882409    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx"
Jul 30 00:56:41.544076 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:41.544045    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp"]
Jul 30 00:56:41.547587 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:41.547567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp"
Jul 30 00:56:41.550204 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:41.550181    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v8q7j\""
Jul 30 00:56:41.554927 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:41.554902    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp"]
Jul 30 00:56:41.620259 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:41.620223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e031e63-4902-4555-aedf-f6a04d1978ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp\" (UID: \"8e031e63-4902-4555-aedf-f6a04d1978ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp"
Jul 30 00:56:41.721027 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:41.720996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e031e63-4902-4555-aedf-f6a04d1978ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp\" (UID: \"8e031e63-4902-4555-aedf-f6a04d1978ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp"
Jul 30 00:56:41.721639 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:41.721616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e031e63-4902-4555-aedf-f6a04d1978ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp\" (UID: \"8e031e63-4902-4555-aedf-f6a04d1978ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp"
Jul 30 00:56:41.860145 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:41.860075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp"
Jul 30 00:56:42.200471 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:42.200443    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp"]
Jul 30 00:56:42.202157 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:56:42.202130    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e031e63_4902_4555_aedf_f6a04d1978ef.slice/crio-e230e1acec3dc2298bd70858b4bc5c3d80cdc53f0b91a87b0e0848ecab126636 WatchSource:0}: Error finding container e230e1acec3dc2298bd70858b4bc5c3d80cdc53f0b91a87b0e0848ecab126636: Status 404 returned error can't find the container with id e230e1acec3dc2298bd70858b4bc5c3d80cdc53f0b91a87b0e0848ecab126636
Jul 30 00:56:42.522534 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:42.522495    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp" event={"ID":"8e031e63-4902-4555-aedf-f6a04d1978ef","Type":"ContainerStarted","Data":"2ca86533a64aeabd6f58c18f227f4407a169e35b1dd829bc16bcdfdb1987556d"}
Jul 30 00:56:42.522674 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:42.522555    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp" event={"ID":"8e031e63-4902-4555-aedf-f6a04d1978ef","Type":"ContainerStarted","Data":"e230e1acec3dc2298bd70858b4bc5c3d80cdc53f0b91a87b0e0848ecab126636"}
Jul 30 00:56:42.539688 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:42.539638    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp" podStartSLOduration=1.539622109 podStartE2EDuration="1.539622109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:56:42.538884178 +0000 UTC m=+2327.836151110" watchObservedRunningTime="2026-07-30 00:56:42.539622109 +0000 UTC m=+2327.836889041"
Jul 30 00:56:43.529000 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:43.528970    2579 generic.go:356] "Generic (PLEG): container finished" podID="8e031e63-4902-4555-aedf-f6a04d1978ef" containerID="2ca86533a64aeabd6f58c18f227f4407a169e35b1dd829bc16bcdfdb1987556d" exitCode=0
Jul 30 00:56:43.529403 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:43.529033    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp" event={"ID":"8e031e63-4902-4555-aedf-f6a04d1978ef","Type":"ContainerDied","Data":"2ca86533a64aeabd6f58c18f227f4407a169e35b1dd829bc16bcdfdb1987556d"}
Jul 30 00:56:44.672860 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:44.672833    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp"
Jul 30 00:56:44.752338 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:44.752310    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e031e63-4902-4555-aedf-f6a04d1978ef-maas-api-service-ca\") pod \"8e031e63-4902-4555-aedf-f6a04d1978ef\" (UID: \"8e031e63-4902-4555-aedf-f6a04d1978ef\") "
Jul 30 00:56:44.752668 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:44.752648    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e031e63-4902-4555-aedf-f6a04d1978ef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8e031e63-4902-4555-aedf-f6a04d1978ef" (UID: "8e031e63-4902-4555-aedf-f6a04d1978ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:56:44.853239 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:44.853173    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e031e63-4902-4555-aedf-f6a04d1978ef-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:56:45.540979 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:45.540948    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp"
Jul 30 00:56:45.541142 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:45.540993    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp" event={"ID":"8e031e63-4902-4555-aedf-f6a04d1978ef","Type":"ContainerDied","Data":"e230e1acec3dc2298bd70858b4bc5c3d80cdc53f0b91a87b0e0848ecab126636"}
Jul 30 00:56:45.541142 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:45.541062    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e230e1acec3dc2298bd70858b4bc5c3d80cdc53f0b91a87b0e0848ecab126636"
Jul 30 00:56:52.028771 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.028580    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"]
Jul 30 00:56:52.029419 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.029384    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc" podUID="d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a" containerName="maas-api" containerID="cri-o://81811cca3f080682c0c68d60dca8fc87df419ee608e89f37a7e6f0b7cca9c67d" gracePeriod=30
Jul 30 00:56:52.083910 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.083880    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw"]
Jul 30 00:56:52.084364 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.084098    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw" podUID="13afd1e6-cd05-489f-acd3-f4f907acbbb5" containerName="payload-processing" containerID="cri-o://b2d88a9960930ddd26acd6c1693be7c54d1718e8a85d01390ff67339172ff8eb" gracePeriod=30
Jul 30 00:56:52.091055 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.091027    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2"]
Jul 30 00:56:52.091503 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.091278    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2" podUID="9208c47e-7c14-4f3b-9076-7fe33aec43d2" containerName="payload-pre-processing" containerID="cri-o://7c20265d69736a17f480e63cd496fe1283c66dcd68e6952d30922393a34aa1d5" gracePeriod=30
Jul 30 00:56:52.222284 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.222208    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp"]
Jul 30 00:56:52.224599 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.224571    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2239f6-m2nfp"]
Jul 30 00:56:52.388003 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.387975    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw"
Jul 30 00:56:52.390412 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.390391    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2"
Jul 30 00:56:52.466009 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.465986    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"
Jul 30 00:56:52.523305 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.523271    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7gqj\" (UniqueName: \"kubernetes.io/projected/9208c47e-7c14-4f3b-9076-7fe33aec43d2-kube-api-access-r7gqj\") pod \"9208c47e-7c14-4f3b-9076-7fe33aec43d2\" (UID: \"9208c47e-7c14-4f3b-9076-7fe33aec43d2\") "
Jul 30 00:56:52.523497 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.523322    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9208c47e-7c14-4f3b-9076-7fe33aec43d2-plugins-config-volume\") pod \"9208c47e-7c14-4f3b-9076-7fe33aec43d2\" (UID: \"9208c47e-7c14-4f3b-9076-7fe33aec43d2\") "
Jul 30 00:56:52.523497 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.523432    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13afd1e6-cd05-489f-acd3-f4f907acbbb5-plugins-config-volume\") pod \"13afd1e6-cd05-489f-acd3-f4f907acbbb5\" (UID: \"13afd1e6-cd05-489f-acd3-f4f907acbbb5\") "
Jul 30 00:56:52.523497 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.523486    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ckjs\" (UniqueName: \"kubernetes.io/projected/13afd1e6-cd05-489f-acd3-f4f907acbbb5-kube-api-access-9ckjs\") pod \"13afd1e6-cd05-489f-acd3-f4f907acbbb5\" (UID: \"13afd1e6-cd05-489f-acd3-f4f907acbbb5\") "
Jul 30 00:56:52.523756 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.523726    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9208c47e-7c14-4f3b-9076-7fe33aec43d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9208c47e-7c14-4f3b-9076-7fe33aec43d2" (UID: "9208c47e-7c14-4f3b-9076-7fe33aec43d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:56:52.523809 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.523762    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13afd1e6-cd05-489f-acd3-f4f907acbbb5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13afd1e6-cd05-489f-acd3-f4f907acbbb5" (UID: "13afd1e6-cd05-489f-acd3-f4f907acbbb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:56:52.525504 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.525479    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9208c47e-7c14-4f3b-9076-7fe33aec43d2-kube-api-access-r7gqj" (OuterVolumeSpecName: "kube-api-access-r7gqj") pod "9208c47e-7c14-4f3b-9076-7fe33aec43d2" (UID: "9208c47e-7c14-4f3b-9076-7fe33aec43d2"). InnerVolumeSpecName "kube-api-access-r7gqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:56:52.525504 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.525507    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13afd1e6-cd05-489f-acd3-f4f907acbbb5-kube-api-access-9ckjs" (OuterVolumeSpecName: "kube-api-access-9ckjs") pod "13afd1e6-cd05-489f-acd3-f4f907acbbb5" (UID: "13afd1e6-cd05-489f-acd3-f4f907acbbb5"). InnerVolumeSpecName "kube-api-access-9ckjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:56:52.583136 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.583062    2579 generic.go:356] "Generic (PLEG): container finished" podID="13afd1e6-cd05-489f-acd3-f4f907acbbb5" containerID="b2d88a9960930ddd26acd6c1693be7c54d1718e8a85d01390ff67339172ff8eb" exitCode=0
Jul 30 00:56:52.583136 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.583103    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw"
Jul 30 00:56:52.583353 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.583146    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw" event={"ID":"13afd1e6-cd05-489f-acd3-f4f907acbbb5","Type":"ContainerDied","Data":"b2d88a9960930ddd26acd6c1693be7c54d1718e8a85d01390ff67339172ff8eb"}
Jul 30 00:56:52.583353 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.583191    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw" event={"ID":"13afd1e6-cd05-489f-acd3-f4f907acbbb5","Type":"ContainerDied","Data":"0977668348fe2fc691842d519fa6b5e56abff07f6dfa789d9b9b1f24f0934e44"}
Jul 30 00:56:52.583353 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.583204    2579 scope.go:117] "RemoveContainer" containerID="b2d88a9960930ddd26acd6c1693be7c54d1718e8a85d01390ff67339172ff8eb"
Jul 30 00:56:52.584534 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.584512    2579 generic.go:356] "Generic (PLEG): container finished" podID="9208c47e-7c14-4f3b-9076-7fe33aec43d2" containerID="7c20265d69736a17f480e63cd496fe1283c66dcd68e6952d30922393a34aa1d5" exitCode=0
Jul 30 00:56:52.584655 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.584549    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2" event={"ID":"9208c47e-7c14-4f3b-9076-7fe33aec43d2","Type":"ContainerDied","Data":"7c20265d69736a17f480e63cd496fe1283c66dcd68e6952d30922393a34aa1d5"}
Jul 30 00:56:52.584655 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.584568    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2"
Jul 30 00:56:52.584655 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.584590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2" event={"ID":"9208c47e-7c14-4f3b-9076-7fe33aec43d2","Type":"ContainerDied","Data":"e68cc96bd0bdc26d493698379c7f3aafedd370cecf50ad89bf71fd5e2e0b631b"}
Jul 30 00:56:52.585939 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.585915    2579 generic.go:356] "Generic (PLEG): container finished" podID="d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a" containerID="81811cca3f080682c0c68d60dca8fc87df419ee608e89f37a7e6f0b7cca9c67d" exitCode=0
Jul 30 00:56:52.586044 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.585952    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"
Jul 30 00:56:52.586044 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.585956    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc" event={"ID":"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a","Type":"ContainerDied","Data":"81811cca3f080682c0c68d60dca8fc87df419ee608e89f37a7e6f0b7cca9c67d"}
Jul 30 00:56:52.586128 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.586064    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc" event={"ID":"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a","Type":"ContainerDied","Data":"5fb301290808e544dc1019ca425a46043c2d9d4d626e344fd1f392ac27dfbba4"}
Jul 30 00:56:52.592373 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.592322    2579 scope.go:117] "RemoveContainer" containerID="b2d88a9960930ddd26acd6c1693be7c54d1718e8a85d01390ff67339172ff8eb"
Jul 30 00:56:52.592602 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:56:52.592584    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d88a9960930ddd26acd6c1693be7c54d1718e8a85d01390ff67339172ff8eb\": container with ID starting with b2d88a9960930ddd26acd6c1693be7c54d1718e8a85d01390ff67339172ff8eb not found: ID does not exist" containerID="b2d88a9960930ddd26acd6c1693be7c54d1718e8a85d01390ff67339172ff8eb"
Jul 30 00:56:52.592658 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.592610    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d88a9960930ddd26acd6c1693be7c54d1718e8a85d01390ff67339172ff8eb"} err="failed to get container status \"b2d88a9960930ddd26acd6c1693be7c54d1718e8a85d01390ff67339172ff8eb\": rpc error: code = NotFound desc = could not find container \"b2d88a9960930ddd26acd6c1693be7c54d1718e8a85d01390ff67339172ff8eb\": container with ID starting with b2d88a9960930ddd26acd6c1693be7c54d1718e8a85d01390ff67339172ff8eb not found: ID does not exist"
Jul 30 00:56:52.592658 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.592627    2579 scope.go:117] "RemoveContainer" containerID="7c20265d69736a17f480e63cd496fe1283c66dcd68e6952d30922393a34aa1d5"
Jul 30 00:56:52.600933 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.600918    2579 scope.go:117] "RemoveContainer" containerID="7c20265d69736a17f480e63cd496fe1283c66dcd68e6952d30922393a34aa1d5"
Jul 30 00:56:52.601161 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:56:52.601142    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c20265d69736a17f480e63cd496fe1283c66dcd68e6952d30922393a34aa1d5\": container with ID starting with 7c20265d69736a17f480e63cd496fe1283c66dcd68e6952d30922393a34aa1d5 not found: ID does not exist" containerID="7c20265d69736a17f480e63cd496fe1283c66dcd68e6952d30922393a34aa1d5"
Jul 30 00:56:52.601208 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.601167    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c20265d69736a17f480e63cd496fe1283c66dcd68e6952d30922393a34aa1d5"} err="failed to get container status \"7c20265d69736a17f480e63cd496fe1283c66dcd68e6952d30922393a34aa1d5\": rpc error: code = NotFound desc = could not find container \"7c20265d69736a17f480e63cd496fe1283c66dcd68e6952d30922393a34aa1d5\": container with ID starting with 7c20265d69736a17f480e63cd496fe1283c66dcd68e6952d30922393a34aa1d5 not found: ID does not exist"
Jul 30 00:56:52.601208 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.601181    2579 scope.go:117] "RemoveContainer" containerID="81811cca3f080682c0c68d60dca8fc87df419ee608e89f37a7e6f0b7cca9c67d"
Jul 30 00:56:52.609292 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.609273    2579 scope.go:117] "RemoveContainer" containerID="81811cca3f080682c0c68d60dca8fc87df419ee608e89f37a7e6f0b7cca9c67d"
Jul 30 00:56:52.609584 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:56:52.609565    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81811cca3f080682c0c68d60dca8fc87df419ee608e89f37a7e6f0b7cca9c67d\": container with ID starting with 81811cca3f080682c0c68d60dca8fc87df419ee608e89f37a7e6f0b7cca9c67d not found: ID does not exist" containerID="81811cca3f080682c0c68d60dca8fc87df419ee608e89f37a7e6f0b7cca9c67d"
Jul 30 00:56:52.609633 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.609590    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81811cca3f080682c0c68d60dca8fc87df419ee608e89f37a7e6f0b7cca9c67d"} err="failed to get container status \"81811cca3f080682c0c68d60dca8fc87df419ee608e89f37a7e6f0b7cca9c67d\": rpc error: code = NotFound desc = could not find container \"81811cca3f080682c0c68d60dca8fc87df419ee608e89f37a7e6f0b7cca9c67d\": container with ID starting with 81811cca3f080682c0c68d60dca8fc87df419ee608e89f37a7e6f0b7cca9c67d not found: ID does not exist"
Jul 30 00:56:52.609697 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.609682    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw"]
Jul 30 00:56:52.613190 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.613168    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2239f6-7856649d7d-6lscw"]
Jul 30 00:56:52.624116 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.624094    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2"]
Jul 30 00:56:52.624430 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.624411    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-maas-api-tls\") pod \"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a\" (UID: \"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a\") "
Jul 30 00:56:52.624521 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.624498    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-openshift-service-ca\") pod \"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a\" (UID: \"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a\") "
Jul 30 00:56:52.624638 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.624542    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxghx\" (UniqueName: \"kubernetes.io/projected/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-kube-api-access-cxghx\") pod \"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a\" (UID: \"d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a\") "
Jul 30 00:56:52.624818 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.624800    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9208c47e-7c14-4f3b-9076-7fe33aec43d2-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:56:52.624878 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.624811    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a" (UID: "d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:56:52.624878 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.624823    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13afd1e6-cd05-489f-acd3-f4f907acbbb5-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:56:52.624878 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.624837    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ckjs\" (UniqueName: \"kubernetes.io/projected/13afd1e6-cd05-489f-acd3-f4f907acbbb5-kube-api-access-9ckjs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:56:52.624878 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.624850    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7gqj\" (UniqueName: \"kubernetes.io/projected/9208c47e-7c14-4f3b-9076-7fe33aec43d2-kube-api-access-r7gqj\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:56:52.626724 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.626695    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a" (UID: "d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:56:52.626812 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.626773    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-kube-api-access-cxghx" (OuterVolumeSpecName: "kube-api-access-cxghx") pod "d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a" (UID: "d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a"). InnerVolumeSpecName "kube-api-access-cxghx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:56:52.628264 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.628227    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2239f6-9cc599656-8lzv2"]
Jul 30 00:56:52.725435 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.725390    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:56:52.725435 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.725430    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:56:52.725435 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.725441    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxghx\" (UniqueName: \"kubernetes.io/projected/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a-kube-api-access-cxghx\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:56:52.909276 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.909231    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"]
Jul 30 00:56:52.913568 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:52.913537    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2239f6-65d6c6fdc6-lqmrc"]
Jul 30 00:56:53.322622 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:53.322543    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13afd1e6-cd05-489f-acd3-f4f907acbbb5" path="/var/lib/kubelet/pods/13afd1e6-cd05-489f-acd3-f4f907acbbb5/volumes"
Jul 30 00:56:53.322966 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:53.322873    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e031e63-4902-4555-aedf-f6a04d1978ef" path="/var/lib/kubelet/pods/8e031e63-4902-4555-aedf-f6a04d1978ef/volumes"
Jul 30 00:56:53.323182 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:53.323170    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9208c47e-7c14-4f3b-9076-7fe33aec43d2" path="/var/lib/kubelet/pods/9208c47e-7c14-4f3b-9076-7fe33aec43d2/volumes"
Jul 30 00:56:53.323567 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:53.323554    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a" path="/var/lib/kubelet/pods/d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a/volumes"
Jul 30 00:56:55.538555 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:55.538524    2579 scope.go:117] "RemoveContainer" containerID="9469025ecbea3d45eecec916847757b96febaedd8fda5c52ffe6a4ef41b3c550"
Jul 30 00:56:59.059978 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.059942    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6898ea-fghm7"]
Jul 30 00:56:59.060369 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.060361    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9208c47e-7c14-4f3b-9076-7fe33aec43d2" containerName="payload-pre-processing"
Jul 30 00:56:59.060413 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.060373    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9208c47e-7c14-4f3b-9076-7fe33aec43d2" containerName="payload-pre-processing"
Jul 30 00:56:59.060413 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.060383    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13afd1e6-cd05-489f-acd3-f4f907acbbb5" containerName="payload-processing"
Jul 30 00:56:59.060413 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.060389    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="13afd1e6-cd05-489f-acd3-f4f907acbbb5" containerName="payload-processing"
Jul 30 00:56:59.060413 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.060396    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a" containerName="maas-api"
Jul 30 00:56:59.060413 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.060402    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a" containerName="maas-api"
Jul 30 00:56:59.060413 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.060410    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e031e63-4902-4555-aedf-f6a04d1978ef" containerName="revoke-keys"
Jul 30 00:56:59.060413 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.060415    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e031e63-4902-4555-aedf-f6a04d1978ef" containerName="revoke-keys"
Jul 30 00:56:59.060630 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.060479    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="13afd1e6-cd05-489f-acd3-f4f907acbbb5" containerName="payload-processing"
Jul 30 00:56:59.060630 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.060491    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9b93ceb-7ca2-4e68-b331-2c2bbc66e51a" containerName="maas-api"
Jul 30 00:56:59.060630 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.060497    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e031e63-4902-4555-aedf-f6a04d1978ef" containerName="revoke-keys"
Jul 30 00:56:59.060630 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.060504    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9208c47e-7c14-4f3b-9076-7fe33aec43d2" containerName="payload-pre-processing"
Jul 30 00:56:59.065532 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.065515    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6898ea-fghm7"
Jul 30 00:56:59.068382 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.068359    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v8q7j\""
Jul 30 00:56:59.070314 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.070284    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6898ea-fghm7"]
Jul 30 00:56:59.087035 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.087009    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/882d70e5-50ca-494b-a10e-18140a8f3992-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6898ea-fghm7\" (UID: \"882d70e5-50ca-494b-a10e-18140a8f3992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6898ea-fghm7"
Jul 30 00:56:59.188479 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.188449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/882d70e5-50ca-494b-a10e-18140a8f3992-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6898ea-fghm7\" (UID: \"882d70e5-50ca-494b-a10e-18140a8f3992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6898ea-fghm7"
Jul 30 00:56:59.189070 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.189051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/882d70e5-50ca-494b-a10e-18140a8f3992-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6898ea-fghm7\" (UID: \"882d70e5-50ca-494b-a10e-18140a8f3992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6898ea-fghm7"
Jul 30 00:56:59.379636 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.379604    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6898ea-fghm7"
Jul 30 00:56:59.517925 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.517900    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6898ea-fghm7"]
Jul 30 00:56:59.520632 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:56:59.520602    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882d70e5_50ca_494b_a10e_18140a8f3992.slice/crio-3e679d12f5ed6a70bb33c85a05f1040ab81bbfb52a32649d21216bb4d7aa09e4 WatchSource:0}: Error finding container 3e679d12f5ed6a70bb33c85a05f1040ab81bbfb52a32649d21216bb4d7aa09e4: Status 404 returned error can't find the container with id 3e679d12f5ed6a70bb33c85a05f1040ab81bbfb52a32649d21216bb4d7aa09e4
Jul 30 00:56:59.627799 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:56:59.627767    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6898ea-fghm7" event={"ID":"882d70e5-50ca-494b-a10e-18140a8f3992","Type":"ContainerStarted","Data":"3e679d12f5ed6a70bb33c85a05f1040ab81bbfb52a32649d21216bb4d7aa09e4"}
Jul 30 00:57:00.634156 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:00.634124    2579 generic.go:356] "Generic (PLEG): container finished" podID="882d70e5-50ca-494b-a10e-18140a8f3992" containerID="945146d373fefc7e7f623271ad277ec9db85cbbf39d2e8eb59c4fe29300fbbbf" exitCode=0
Jul 30 00:57:00.634536 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:00.634213    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6898ea-fghm7" event={"ID":"882d70e5-50ca-494b-a10e-18140a8f3992","Type":"ContainerDied","Data":"945146d373fefc7e7f623271ad277ec9db85cbbf39d2e8eb59c4fe29300fbbbf"}
Jul 30 00:57:01.772428 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:01.772404    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6898ea-fghm7"
Jul 30 00:57:01.813313 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:01.813277    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/882d70e5-50ca-494b-a10e-18140a8f3992-maas-api-service-ca\") pod \"882d70e5-50ca-494b-a10e-18140a8f3992\" (UID: \"882d70e5-50ca-494b-a10e-18140a8f3992\") "
Jul 30 00:57:01.813641 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:01.813619    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882d70e5-50ca-494b-a10e-18140a8f3992-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "882d70e5-50ca-494b-a10e-18140a8f3992" (UID: "882d70e5-50ca-494b-a10e-18140a8f3992"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:57:01.914711 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:01.914631    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/882d70e5-50ca-494b-a10e-18140a8f3992-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:57:02.647011 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:02.646983    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6898ea-fghm7"
Jul 30 00:57:02.647193 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:02.646981    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6898ea-fghm7" event={"ID":"882d70e5-50ca-494b-a10e-18140a8f3992","Type":"ContainerDied","Data":"3e679d12f5ed6a70bb33c85a05f1040ab81bbfb52a32649d21216bb4d7aa09e4"}
Jul 30 00:57:02.647193 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:02.647109    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e679d12f5ed6a70bb33c85a05f1040ab81bbfb52a32649d21216bb4d7aa09e4"
Jul 30 00:57:09.494930 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.494897    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"]
Jul 30 00:57:09.495361 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.495118    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx" podUID="299ca293-6666-4d2a-bddd-624afdf6be85" containerName="maas-api" containerID="cri-o://c8fd637660bcfa812af7ea2db37ded4f201fb275c8988369a7797c3fb01d2e53" gracePeriod=30
Jul 30 00:57:09.587786 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.587756    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8"]
Jul 30 00:57:09.587982 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.587958    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8" podUID="29fe2a57-d801-4c9b-bd73-caaf8013f94d" containerName="payload-processing" containerID="cri-o://190d9ec577b3f0f3780c7149135aee860b37704fcc2ea72b12253c3722f612ea" gracePeriod=30
Jul 30 00:57:09.599641 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.599601    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx"]
Jul 30 00:57:09.600143 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.600114    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx" podUID="79857008-e5dc-488b-809a-fb547fc88bdc" containerName="payload-pre-processing" containerID="cri-o://5dbc7be15924baaa8e6b9c438f9ba040538243cc72c9e7db15fb167eeabb6422" gracePeriod=30
Jul 30 00:57:09.686232 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.686043    2579 generic.go:356] "Generic (PLEG): container finished" podID="299ca293-6666-4d2a-bddd-624afdf6be85" containerID="c8fd637660bcfa812af7ea2db37ded4f201fb275c8988369a7797c3fb01d2e53" exitCode=0
Jul 30 00:57:09.686232 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.686116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx" event={"ID":"299ca293-6666-4d2a-bddd-624afdf6be85","Type":"ContainerDied","Data":"c8fd637660bcfa812af7ea2db37ded4f201fb275c8988369a7797c3fb01d2e53"}
Jul 30 00:57:09.725919 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.725882    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6898ea-fghm7"]
Jul 30 00:57:09.727754 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.727728    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6898ea-fghm7"]
Jul 30 00:57:09.775396 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.775373    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"
Jul 30 00:57:09.870287 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.870264    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx"
Jul 30 00:57:09.873533 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.873512    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8"
Jul 30 00:57:09.880874 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.880842    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n452x\" (UniqueName: \"kubernetes.io/projected/299ca293-6666-4d2a-bddd-624afdf6be85-kube-api-access-n452x\") pod \"299ca293-6666-4d2a-bddd-624afdf6be85\" (UID: \"299ca293-6666-4d2a-bddd-624afdf6be85\") "
Jul 30 00:57:09.880958 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.880884    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/299ca293-6666-4d2a-bddd-624afdf6be85-openshift-service-ca\") pod \"299ca293-6666-4d2a-bddd-624afdf6be85\" (UID: \"299ca293-6666-4d2a-bddd-624afdf6be85\") "
Jul 30 00:57:09.880958 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.880947    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/299ca293-6666-4d2a-bddd-624afdf6be85-maas-api-tls\") pod \"299ca293-6666-4d2a-bddd-624afdf6be85\" (UID: \"299ca293-6666-4d2a-bddd-624afdf6be85\") "
Jul 30 00:57:09.881220 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.881198    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/299ca293-6666-4d2a-bddd-624afdf6be85-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "299ca293-6666-4d2a-bddd-624afdf6be85" (UID: "299ca293-6666-4d2a-bddd-624afdf6be85"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:57:09.882918 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.882896    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299ca293-6666-4d2a-bddd-624afdf6be85-kube-api-access-n452x" (OuterVolumeSpecName: "kube-api-access-n452x") pod "299ca293-6666-4d2a-bddd-624afdf6be85" (UID: "299ca293-6666-4d2a-bddd-624afdf6be85"). InnerVolumeSpecName "kube-api-access-n452x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:57:09.882918 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.882909    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/299ca293-6666-4d2a-bddd-624afdf6be85-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "299ca293-6666-4d2a-bddd-624afdf6be85" (UID: "299ca293-6666-4d2a-bddd-624afdf6be85"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:57:09.982074 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.982049    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mxck\" (UniqueName: \"kubernetes.io/projected/79857008-e5dc-488b-809a-fb547fc88bdc-kube-api-access-5mxck\") pod \"79857008-e5dc-488b-809a-fb547fc88bdc\" (UID: \"79857008-e5dc-488b-809a-fb547fc88bdc\") "
Jul 30 00:57:09.982284 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.982086    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29fe2a57-d801-4c9b-bd73-caaf8013f94d-plugins-config-volume\") pod \"29fe2a57-d801-4c9b-bd73-caaf8013f94d\" (UID: \"29fe2a57-d801-4c9b-bd73-caaf8013f94d\") "
Jul 30 00:57:09.982284 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.982109    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79857008-e5dc-488b-809a-fb547fc88bdc-plugins-config-volume\") pod \"79857008-e5dc-488b-809a-fb547fc88bdc\" (UID: \"79857008-e5dc-488b-809a-fb547fc88bdc\") "
Jul 30 00:57:09.982284 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.982188    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sgr9\" (UniqueName: \"kubernetes.io/projected/29fe2a57-d801-4c9b-bd73-caaf8013f94d-kube-api-access-4sgr9\") pod \"29fe2a57-d801-4c9b-bd73-caaf8013f94d\" (UID: \"29fe2a57-d801-4c9b-bd73-caaf8013f94d\") "
Jul 30 00:57:09.982478 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.982379    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n452x\" (UniqueName: \"kubernetes.io/projected/299ca293-6666-4d2a-bddd-624afdf6be85-kube-api-access-n452x\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:57:09.982478 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.982392    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/299ca293-6666-4d2a-bddd-624afdf6be85-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:57:09.982478 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.982402    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/299ca293-6666-4d2a-bddd-624afdf6be85-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:57:09.982615 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.982528    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79857008-e5dc-488b-809a-fb547fc88bdc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "79857008-e5dc-488b-809a-fb547fc88bdc" (UID: "79857008-e5dc-488b-809a-fb547fc88bdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:57:09.982615 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.982526    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29fe2a57-d801-4c9b-bd73-caaf8013f94d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29fe2a57-d801-4c9b-bd73-caaf8013f94d" (UID: "29fe2a57-d801-4c9b-bd73-caaf8013f94d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:57:09.984192 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.984174    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79857008-e5dc-488b-809a-fb547fc88bdc-kube-api-access-5mxck" (OuterVolumeSpecName: "kube-api-access-5mxck") pod "79857008-e5dc-488b-809a-fb547fc88bdc" (UID: "79857008-e5dc-488b-809a-fb547fc88bdc"). InnerVolumeSpecName "kube-api-access-5mxck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:57:09.984273 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:09.984212    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29fe2a57-d801-4c9b-bd73-caaf8013f94d-kube-api-access-4sgr9" (OuterVolumeSpecName: "kube-api-access-4sgr9") pod "29fe2a57-d801-4c9b-bd73-caaf8013f94d" (UID: "29fe2a57-d801-4c9b-bd73-caaf8013f94d"). InnerVolumeSpecName "kube-api-access-4sgr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:57:10.083311 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.083223    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mxck\" (UniqueName: \"kubernetes.io/projected/79857008-e5dc-488b-809a-fb547fc88bdc-kube-api-access-5mxck\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:57:10.083311 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.083268    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29fe2a57-d801-4c9b-bd73-caaf8013f94d-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:57:10.083311 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.083277    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79857008-e5dc-488b-809a-fb547fc88bdc-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:57:10.083311 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.083285    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sgr9\" (UniqueName: \"kubernetes.io/projected/29fe2a57-d801-4c9b-bd73-caaf8013f94d-kube-api-access-4sgr9\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:57:10.692233 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.692202    2579 generic.go:356] "Generic (PLEG): container finished" podID="79857008-e5dc-488b-809a-fb547fc88bdc" containerID="5dbc7be15924baaa8e6b9c438f9ba040538243cc72c9e7db15fb167eeabb6422" exitCode=0
Jul 30 00:57:10.692691 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.692259    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx"
Jul 30 00:57:10.692691 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.692278    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx" event={"ID":"79857008-e5dc-488b-809a-fb547fc88bdc","Type":"ContainerDied","Data":"5dbc7be15924baaa8e6b9c438f9ba040538243cc72c9e7db15fb167eeabb6422"}
Jul 30 00:57:10.692691 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.692328    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx" event={"ID":"79857008-e5dc-488b-809a-fb547fc88bdc","Type":"ContainerDied","Data":"c427ca6949924a5cacf2ff510bf24bce4bf35039df78248dbf3d1a9ffa7e758a"}
Jul 30 00:57:10.692691 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.692355    2579 scope.go:117] "RemoveContainer" containerID="5dbc7be15924baaa8e6b9c438f9ba040538243cc72c9e7db15fb167eeabb6422"
Jul 30 00:57:10.693714 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.693700    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"
Jul 30 00:57:10.693795 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.693719    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx" event={"ID":"299ca293-6666-4d2a-bddd-624afdf6be85","Type":"ContainerDied","Data":"2cc38a49d1c522ef8cf0bb3bbcb47d6a4eab561893ad984fd7aa30a6ada9074a"}
Jul 30 00:57:10.695107 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.695061    2579 generic.go:356] "Generic (PLEG): container finished" podID="29fe2a57-d801-4c9b-bd73-caaf8013f94d" containerID="190d9ec577b3f0f3780c7149135aee860b37704fcc2ea72b12253c3722f612ea" exitCode=0
Jul 30 00:57:10.695221 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.695111    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8"
Jul 30 00:57:10.695221 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.695146    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8" event={"ID":"29fe2a57-d801-4c9b-bd73-caaf8013f94d","Type":"ContainerDied","Data":"190d9ec577b3f0f3780c7149135aee860b37704fcc2ea72b12253c3722f612ea"}
Jul 30 00:57:10.695221 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.695177    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8" event={"ID":"29fe2a57-d801-4c9b-bd73-caaf8013f94d","Type":"ContainerDied","Data":"a7d139b1dfdc896355f32b42a65e0731844cf4d41f2e939efe8b60d295792b1f"}
Jul 30 00:57:10.703646 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.703620    2579 scope.go:117] "RemoveContainer" containerID="5dbc7be15924baaa8e6b9c438f9ba040538243cc72c9e7db15fb167eeabb6422"
Jul 30 00:57:10.703934 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:57:10.703913    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dbc7be15924baaa8e6b9c438f9ba040538243cc72c9e7db15fb167eeabb6422\": container with ID starting with 5dbc7be15924baaa8e6b9c438f9ba040538243cc72c9e7db15fb167eeabb6422 not found: ID does not exist" containerID="5dbc7be15924baaa8e6b9c438f9ba040538243cc72c9e7db15fb167eeabb6422"
Jul 30 00:57:10.703995 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.703943    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dbc7be15924baaa8e6b9c438f9ba040538243cc72c9e7db15fb167eeabb6422"} err="failed to get container status \"5dbc7be15924baaa8e6b9c438f9ba040538243cc72c9e7db15fb167eeabb6422\": rpc error: code = NotFound desc = could not find container \"5dbc7be15924baaa8e6b9c438f9ba040538243cc72c9e7db15fb167eeabb6422\": container with ID starting with 5dbc7be15924baaa8e6b9c438f9ba040538243cc72c9e7db15fb167eeabb6422 not found: ID does not exist"
Jul 30 00:57:10.703995 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.703960    2579 scope.go:117] "RemoveContainer" containerID="c8fd637660bcfa812af7ea2db37ded4f201fb275c8988369a7797c3fb01d2e53"
Jul 30 00:57:10.712013 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.711998    2579 scope.go:117] "RemoveContainer" containerID="190d9ec577b3f0f3780c7149135aee860b37704fcc2ea72b12253c3722f612ea"
Jul 30 00:57:10.719964 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.719938    2579 scope.go:117] "RemoveContainer" containerID="190d9ec577b3f0f3780c7149135aee860b37704fcc2ea72b12253c3722f612ea"
Jul 30 00:57:10.720667 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:57:10.720281    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"190d9ec577b3f0f3780c7149135aee860b37704fcc2ea72b12253c3722f612ea\": container with ID starting with 190d9ec577b3f0f3780c7149135aee860b37704fcc2ea72b12253c3722f612ea not found: ID does not exist" containerID="190d9ec577b3f0f3780c7149135aee860b37704fcc2ea72b12253c3722f612ea"
Jul 30 00:57:10.720667 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.720314    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"190d9ec577b3f0f3780c7149135aee860b37704fcc2ea72b12253c3722f612ea"} err="failed to get container status \"190d9ec577b3f0f3780c7149135aee860b37704fcc2ea72b12253c3722f612ea\": rpc error: code = NotFound desc = could not find container \"190d9ec577b3f0f3780c7149135aee860b37704fcc2ea72b12253c3722f612ea\": container with ID starting with 190d9ec577b3f0f3780c7149135aee860b37704fcc2ea72b12253c3722f612ea not found: ID does not exist"
Jul 30 00:57:10.724131 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.724109    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx"]
Jul 30 00:57:10.728336 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.728309    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6898ea-54ff7589b-sjjpx"]
Jul 30 00:57:10.738293 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.738270    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"]
Jul 30 00:57:10.742013 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.741991    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6898ea-647995bc98-c88qx"]
Jul 30 00:57:10.751641 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.751616    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8"]
Jul 30 00:57:10.756944 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:10.756922    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6898ea-564c47cb6-57nr8"]
Jul 30 00:57:11.322501 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:11.322463    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="299ca293-6666-4d2a-bddd-624afdf6be85" path="/var/lib/kubelet/pods/299ca293-6666-4d2a-bddd-624afdf6be85/volumes"
Jul 30 00:57:11.322896 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:11.322876    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29fe2a57-d801-4c9b-bd73-caaf8013f94d" path="/var/lib/kubelet/pods/29fe2a57-d801-4c9b-bd73-caaf8013f94d/volumes"
Jul 30 00:57:11.323279 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:11.323262    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79857008-e5dc-488b-809a-fb547fc88bdc" path="/var/lib/kubelet/pods/79857008-e5dc-488b-809a-fb547fc88bdc/volumes"
Jul 30 00:57:11.323617 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:11.323601    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="882d70e5-50ca-494b-a10e-18140a8f3992" path="/var/lib/kubelet/pods/882d70e5-50ca-494b-a10e-18140a8f3992/volumes"
Jul 30 00:57:20.624273 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.624227    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"]
Jul 30 00:57:20.624693 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.624647    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="299ca293-6666-4d2a-bddd-624afdf6be85" containerName="maas-api"
Jul 30 00:57:20.624693 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.624660    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="299ca293-6666-4d2a-bddd-624afdf6be85" containerName="maas-api"
Jul 30 00:57:20.624693 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.624670    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79857008-e5dc-488b-809a-fb547fc88bdc" containerName="payload-pre-processing"
Jul 30 00:57:20.624693 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.624675    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79857008-e5dc-488b-809a-fb547fc88bdc" containerName="payload-pre-processing"
Jul 30 00:57:20.624693 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.624682    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29fe2a57-d801-4c9b-bd73-caaf8013f94d" containerName="payload-processing"
Jul 30 00:57:20.624693 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.624688    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="29fe2a57-d801-4c9b-bd73-caaf8013f94d" containerName="payload-processing"
Jul 30 00:57:20.624900 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.624699    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="882d70e5-50ca-494b-a10e-18140a8f3992" containerName="revoke-keys"
Jul 30 00:57:20.624900 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.624705    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="882d70e5-50ca-494b-a10e-18140a8f3992" containerName="revoke-keys"
Jul 30 00:57:20.624900 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.624759    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="299ca293-6666-4d2a-bddd-624afdf6be85" containerName="maas-api"
Jul 30 00:57:20.624900 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.624767    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="29fe2a57-d801-4c9b-bd73-caaf8013f94d" containerName="payload-processing"
Jul 30 00:57:20.624900 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.624775    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="882d70e5-50ca-494b-a10e-18140a8f3992" containerName="revoke-keys"
Jul 30 00:57:20.624900 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.624781    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="79857008-e5dc-488b-809a-fb547fc88bdc" containerName="payload-pre-processing"
Jul 30 00:57:20.627811 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.627793    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"
Jul 30 00:57:20.630215 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.630191    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-772c30-serving-cert\""
Jul 30 00:57:20.637966 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.637942    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"]
Jul 30 00:57:20.675574 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.675548    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64acc2b5-ff32-4881-93be-204787352ded-maas-api-tls\") pod \"maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp\" (UID: \"64acc2b5-ff32-4881-93be-204787352ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"
Jul 30 00:57:20.675709 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.675584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64acc2b5-ff32-4881-93be-204787352ded-openshift-service-ca\") pod \"maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp\" (UID: \"64acc2b5-ff32-4881-93be-204787352ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"
Jul 30 00:57:20.675709 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.675627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z69x7\" (UniqueName: \"kubernetes.io/projected/64acc2b5-ff32-4881-93be-204787352ded-kube-api-access-z69x7\") pod \"maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp\" (UID: \"64acc2b5-ff32-4881-93be-204787352ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"
Jul 30 00:57:20.776253 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.776209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64acc2b5-ff32-4881-93be-204787352ded-maas-api-tls\") pod \"maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp\" (UID: \"64acc2b5-ff32-4881-93be-204787352ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"
Jul 30 00:57:20.776253 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.776251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64acc2b5-ff32-4881-93be-204787352ded-openshift-service-ca\") pod \"maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp\" (UID: \"64acc2b5-ff32-4881-93be-204787352ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"
Jul 30 00:57:20.776450 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.776292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z69x7\" (UniqueName: \"kubernetes.io/projected/64acc2b5-ff32-4881-93be-204787352ded-kube-api-access-z69x7\") pod \"maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp\" (UID: \"64acc2b5-ff32-4881-93be-204787352ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"
Jul 30 00:57:20.776450 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:57:20.776385    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-serving-cert: secret "maas-api-e2e-api-a-772c30-serving-cert" not found
Jul 30 00:57:20.776571 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:57:20.776477    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64acc2b5-ff32-4881-93be-204787352ded-maas-api-tls podName:64acc2b5-ff32-4881-93be-204787352ded nodeName:}" failed. No retries permitted until 2026-07-30 00:57:21.276454132 +0000 UTC m=+2366.573721053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/64acc2b5-ff32-4881-93be-204787352ded-maas-api-tls") pod "maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp" (UID: "64acc2b5-ff32-4881-93be-204787352ded") : secret "maas-api-e2e-api-a-772c30-serving-cert" not found
Jul 30 00:57:20.776839 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.776817    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64acc2b5-ff32-4881-93be-204787352ded-openshift-service-ca\") pod \"maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp\" (UID: \"64acc2b5-ff32-4881-93be-204787352ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"
Jul 30 00:57:20.784418 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:20.784401    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z69x7\" (UniqueName: \"kubernetes.io/projected/64acc2b5-ff32-4881-93be-204787352ded-kube-api-access-z69x7\") pod \"maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp\" (UID: \"64acc2b5-ff32-4881-93be-204787352ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"
Jul 30 00:57:21.223864 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.223830    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p"]
Jul 30 00:57:21.227414 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.227398    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p"
Jul 30 00:57:21.230880 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.230858    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-772c30-dockercfg-9xsw9\""
Jul 30 00:57:21.231022 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.230915    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-772c30\""
Jul 30 00:57:21.239709 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.239688    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p"]
Jul 30 00:57:21.263498 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.263474    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt"]
Jul 30 00:57:21.266936 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.266919    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt"
Jul 30 00:57:21.274606 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.274586    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt"]
Jul 30 00:57:21.281622 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.281589    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64acc2b5-ff32-4881-93be-204787352ded-maas-api-tls\") pod \"maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp\" (UID: \"64acc2b5-ff32-4881-93be-204787352ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"
Jul 30 00:57:21.281709 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.281678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmgn8\" (UniqueName: \"kubernetes.io/projected/f04df206-0038-4f0e-b1c3-a4d6f7302e9e-kube-api-access-lmgn8\") pod \"payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p\" (UID: \"f04df206-0038-4f0e-b1c3-a4d6f7302e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p"
Jul 30 00:57:21.281777 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.281712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04df206-0038-4f0e-b1c3-a4d6f7302e9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p\" (UID: \"f04df206-0038-4f0e-b1c3-a4d6f7302e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p"
Jul 30 00:57:21.284210 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.284185    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64acc2b5-ff32-4881-93be-204787352ded-maas-api-tls\") pod \"maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp\" (UID: \"64acc2b5-ff32-4881-93be-204787352ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"
Jul 30 00:57:21.383001 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.382971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9bd2c2-515f-43c6-bee5-804477b13dae-plugins-config-volume\") pod \"payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt\" (UID: \"5f9bd2c2-515f-43c6-bee5-804477b13dae\") " pod="openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt"
Jul 30 00:57:21.383170 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.383025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmgn8\" (UniqueName: \"kubernetes.io/projected/f04df206-0038-4f0e-b1c3-a4d6f7302e9e-kube-api-access-lmgn8\") pod \"payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p\" (UID: \"f04df206-0038-4f0e-b1c3-a4d6f7302e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p"
Jul 30 00:57:21.383170 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.383075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04df206-0038-4f0e-b1c3-a4d6f7302e9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p\" (UID: \"f04df206-0038-4f0e-b1c3-a4d6f7302e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p"
Jul 30 00:57:21.383170 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.383094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sswg\" (UniqueName: \"kubernetes.io/projected/5f9bd2c2-515f-43c6-bee5-804477b13dae-kube-api-access-7sswg\") pod \"payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt\" (UID: \"5f9bd2c2-515f-43c6-bee5-804477b13dae\") " pod="openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt"
Jul 30 00:57:21.383761 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.383742    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04df206-0038-4f0e-b1c3-a4d6f7302e9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p\" (UID: \"f04df206-0038-4f0e-b1c3-a4d6f7302e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p"
Jul 30 00:57:21.395489 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.395460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmgn8\" (UniqueName: \"kubernetes.io/projected/f04df206-0038-4f0e-b1c3-a4d6f7302e9e-kube-api-access-lmgn8\") pod \"payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p\" (UID: \"f04df206-0038-4f0e-b1c3-a4d6f7302e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p"
Jul 30 00:57:21.484690 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.484618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9bd2c2-515f-43c6-bee5-804477b13dae-plugins-config-volume\") pod \"payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt\" (UID: \"5f9bd2c2-515f-43c6-bee5-804477b13dae\") " pod="openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt"
Jul 30 00:57:21.484690 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.484670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sswg\" (UniqueName: \"kubernetes.io/projected/5f9bd2c2-515f-43c6-bee5-804477b13dae-kube-api-access-7sswg\") pod \"payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt\" (UID: \"5f9bd2c2-515f-43c6-bee5-804477b13dae\") " pod="openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt"
Jul 30 00:57:21.485391 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.485371    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9bd2c2-515f-43c6-bee5-804477b13dae-plugins-config-volume\") pod \"payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt\" (UID: \"5f9bd2c2-515f-43c6-bee5-804477b13dae\") " pod="openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt"
Jul 30 00:57:21.493567 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.493547    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sswg\" (UniqueName: \"kubernetes.io/projected/5f9bd2c2-515f-43c6-bee5-804477b13dae-kube-api-access-7sswg\") pod \"payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt\" (UID: \"5f9bd2c2-515f-43c6-bee5-804477b13dae\") " pod="openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt"
Jul 30 00:57:21.539415 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.539376    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p"
Jul 30 00:57:21.541589 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.541569    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"
Jul 30 00:57:21.578540 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.578504    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt"
Jul 30 00:57:21.695678 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.695613    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p"]
Jul 30 00:57:21.697670 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:57:21.697634    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04df206_0038_4f0e_b1c3_a4d6f7302e9e.slice/crio-05696f4c9546a5b344089c87a73f6870dd92802f95abec7cb3119b55300fc4de WatchSource:0}: Error finding container 05696f4c9546a5b344089c87a73f6870dd92802f95abec7cb3119b55300fc4de: Status 404 returned error can't find the container with id 05696f4c9546a5b344089c87a73f6870dd92802f95abec7cb3119b55300fc4de
Jul 30 00:57:21.752307 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.752275    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p" event={"ID":"f04df206-0038-4f0e-b1c3-a4d6f7302e9e","Type":"ContainerStarted","Data":"05696f4c9546a5b344089c87a73f6870dd92802f95abec7cb3119b55300fc4de"}
Jul 30 00:57:21.762084 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.762060    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt"]
Jul 30 00:57:21.763024 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:57:21.762998    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9bd2c2_515f_43c6_bee5_804477b13dae.slice/crio-a7bcd2dfb289b9e4c15935cf1ba3d5bbdc97526671cb1280adb29227cbf266f5 WatchSource:0}: Error finding container a7bcd2dfb289b9e4c15935cf1ba3d5bbdc97526671cb1280adb29227cbf266f5: Status 404 returned error can't find the container with id a7bcd2dfb289b9e4c15935cf1ba3d5bbdc97526671cb1280adb29227cbf266f5
Jul 30 00:57:21.933183 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.933147    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"]
Jul 30 00:57:21.935358 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:57:21.935289    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64acc2b5_ff32_4881_93be_204787352ded.slice/crio-de0e20c7da03a2a03bb9a70054a96d40db60317d7ecba6a393db830959879cea WatchSource:0}: Error finding container de0e20c7da03a2a03bb9a70054a96d40db60317d7ecba6a393db830959879cea: Status 404 returned error can't find the container with id de0e20c7da03a2a03bb9a70054a96d40db60317d7ecba6a393db830959879cea
Jul 30 00:57:21.938017 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:21.937319    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 00:57:22.758620 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:22.758588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p" event={"ID":"f04df206-0038-4f0e-b1c3-a4d6f7302e9e","Type":"ContainerStarted","Data":"101eca073ac245c9bd7c85eb59985fdef05fd7eb122f1229175646abe604499f"}
Jul 30 00:57:22.759102 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:22.758652    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p"
Jul 30 00:57:22.760363 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:22.760332    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt" event={"ID":"5f9bd2c2-515f-43c6-bee5-804477b13dae","Type":"ContainerStarted","Data":"6c56ce4b80931099e0983b5fb6bbfcc1ddf99b226d3751274b253e8adcbd03de"}
Jul 30 00:57:22.760489 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:22.760381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt" event={"ID":"5f9bd2c2-515f-43c6-bee5-804477b13dae","Type":"ContainerStarted","Data":"a7bcd2dfb289b9e4c15935cf1ba3d5bbdc97526671cb1280adb29227cbf266f5"}
Jul 30 00:57:22.760579 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:22.760563    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt"
Jul 30 00:57:22.761687 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:22.761662    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp" event={"ID":"64acc2b5-ff32-4881-93be-204787352ded","Type":"ContainerStarted","Data":"de0e20c7da03a2a03bb9a70054a96d40db60317d7ecba6a393db830959879cea"}
Jul 30 00:57:22.775152 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:22.775107    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p" podStartSLOduration=1.7750917400000001 podStartE2EDuration="1.77509174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:57:22.773991018 +0000 UTC m=+2368.071257962" watchObservedRunningTime="2026-07-30 00:57:22.77509174 +0000 UTC m=+2368.072358709"
Jul 30 00:57:22.789774 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:22.789735    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt" podStartSLOduration=1.789721093 podStartE2EDuration="1.789721093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:57:22.788896217 +0000 UTC m=+2368.086163149" watchObservedRunningTime="2026-07-30 00:57:22.789721093 +0000 UTC m=+2368.086988068"
Jul 30 00:57:23.769219 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:23.769182    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp" event={"ID":"64acc2b5-ff32-4881-93be-204787352ded","Type":"ContainerStarted","Data":"ba68b00bc669b794205e05d02c308b68223ea2a4f683abbe0d4d81bb50b895ad"}
Jul 30 00:57:23.769686 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:23.769262    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"
Jul 30 00:57:23.786288 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:23.786218    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp" podStartSLOduration=2.071891328 podStartE2EDuration="3.786200978s" totalImagesPullingTime="1.71430965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:57:23.785929843 +0000 UTC m=+2369.083196774" watchObservedRunningTime="2026-07-30 00:57:23.786200978 +0000 UTC m=+2369.083467911"
Jul 30 00:57:29.779951 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:29.779924    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"
Jul 30 00:57:33.771433 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:33.771403    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p"
Jul 30 00:57:33.771802 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:33.771628    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt"
Jul 30 00:57:48.907419 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:48.907385    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"]
Jul 30 00:57:48.911190 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:48.911167    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"
Jul 30 00:57:48.913920 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:48.913899    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-68c25c-serving-cert\""
Jul 30 00:57:48.918650 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:48.918625    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"]
Jul 30 00:57:48.929114 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:48.929048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2vf9\" (UniqueName: \"kubernetes.io/projected/b9e10582-b671-4ba8-b4b0-b7d197c60724-kube-api-access-w2vf9\") pod \"maas-api-e2e-api-b-68c25c-f955f9969-zsq4n\" (UID: \"b9e10582-b671-4ba8-b4b0-b7d197c60724\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"
Jul 30 00:57:48.929114 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:48.929103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e10582-b671-4ba8-b4b0-b7d197c60724-openshift-service-ca\") pod \"maas-api-e2e-api-b-68c25c-f955f9969-zsq4n\" (UID: \"b9e10582-b671-4ba8-b4b0-b7d197c60724\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"
Jul 30 00:57:48.929287 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:48.929161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e10582-b671-4ba8-b4b0-b7d197c60724-maas-api-tls\") pod \"maas-api-e2e-api-b-68c25c-f955f9969-zsq4n\" (UID: \"b9e10582-b671-4ba8-b4b0-b7d197c60724\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"
Jul 30 00:57:49.030569 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.030538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e10582-b671-4ba8-b4b0-b7d197c60724-maas-api-tls\") pod \"maas-api-e2e-api-b-68c25c-f955f9969-zsq4n\" (UID: \"b9e10582-b671-4ba8-b4b0-b7d197c60724\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"
Jul 30 00:57:49.030705 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.030641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2vf9\" (UniqueName: \"kubernetes.io/projected/b9e10582-b671-4ba8-b4b0-b7d197c60724-kube-api-access-w2vf9\") pod \"maas-api-e2e-api-b-68c25c-f955f9969-zsq4n\" (UID: \"b9e10582-b671-4ba8-b4b0-b7d197c60724\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"
Jul 30 00:57:49.030705 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.030675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e10582-b671-4ba8-b4b0-b7d197c60724-openshift-service-ca\") pod \"maas-api-e2e-api-b-68c25c-f955f9969-zsq4n\" (UID: \"b9e10582-b671-4ba8-b4b0-b7d197c60724\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"
Jul 30 00:57:49.030705 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:57:49.030688    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-serving-cert: secret "maas-api-e2e-api-b-68c25c-serving-cert" not found
Jul 30 00:57:49.030822 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:57:49.030750    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e10582-b671-4ba8-b4b0-b7d197c60724-maas-api-tls podName:b9e10582-b671-4ba8-b4b0-b7d197c60724 nodeName:}" failed. No retries permitted until 2026-07-30 00:57:49.53073036 +0000 UTC m=+2394.827997284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9e10582-b671-4ba8-b4b0-b7d197c60724-maas-api-tls") pod "maas-api-e2e-api-b-68c25c-f955f9969-zsq4n" (UID: "b9e10582-b671-4ba8-b4b0-b7d197c60724") : secret "maas-api-e2e-api-b-68c25c-serving-cert" not found
Jul 30 00:57:49.031329 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.031312    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e10582-b671-4ba8-b4b0-b7d197c60724-openshift-service-ca\") pod \"maas-api-e2e-api-b-68c25c-f955f9969-zsq4n\" (UID: \"b9e10582-b671-4ba8-b4b0-b7d197c60724\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"
Jul 30 00:57:49.039446 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.039428    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2vf9\" (UniqueName: \"kubernetes.io/projected/b9e10582-b671-4ba8-b4b0-b7d197c60724-kube-api-access-w2vf9\") pod \"maas-api-e2e-api-b-68c25c-f955f9969-zsq4n\" (UID: \"b9e10582-b671-4ba8-b4b0-b7d197c60724\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"
Jul 30 00:57:49.503239 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.503199    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn"]
Jul 30 00:57:49.507260 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.507225    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn"
Jul 30 00:57:49.510023 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.510000    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-68c25c\""
Jul 30 00:57:49.516052 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.516027    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn"]
Jul 30 00:57:49.537427 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.537395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/857ae7f8-2279-431c-9ae2-81844ba85c56-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn\" (UID: \"857ae7f8-2279-431c-9ae2-81844ba85c56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn"
Jul 30 00:57:49.537538 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.537522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e10582-b671-4ba8-b4b0-b7d197c60724-maas-api-tls\") pod \"maas-api-e2e-api-b-68c25c-f955f9969-zsq4n\" (UID: \"b9e10582-b671-4ba8-b4b0-b7d197c60724\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"
Jul 30 00:57:49.537620 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.537604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqtks\" (UniqueName: \"kubernetes.io/projected/857ae7f8-2279-431c-9ae2-81844ba85c56-kube-api-access-tqtks\") pod \"payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn\" (UID: \"857ae7f8-2279-431c-9ae2-81844ba85c56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn"
Jul 30 00:57:49.539795 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.539769    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e10582-b671-4ba8-b4b0-b7d197c60724-maas-api-tls\") pod \"maas-api-e2e-api-b-68c25c-f955f9969-zsq4n\" (UID: \"b9e10582-b671-4ba8-b4b0-b7d197c60724\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"
Jul 30 00:57:49.548178 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.548158    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5"]
Jul 30 00:57:49.551773 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.551758    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5"
Jul 30 00:57:49.554415 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.554398    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-68c25c-dockercfg-p5d56\""
Jul 30 00:57:49.560641 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.560619    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5"]
Jul 30 00:57:49.638597 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.638569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v8sj\" (UniqueName: \"kubernetes.io/projected/79d43c8e-cde1-45c4-9230-b50f062ec0d7-kube-api-access-5v8sj\") pod \"payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5\" (UID: \"79d43c8e-cde1-45c4-9230-b50f062ec0d7\") " pod="openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5"
Jul 30 00:57:49.638737 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.638607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d43c8e-cde1-45c4-9230-b50f062ec0d7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5\" (UID: \"79d43c8e-cde1-45c4-9230-b50f062ec0d7\") " pod="openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5"
Jul 30 00:57:49.638793 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.638770    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqtks\" (UniqueName: \"kubernetes.io/projected/857ae7f8-2279-431c-9ae2-81844ba85c56-kube-api-access-tqtks\") pod \"payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn\" (UID: \"857ae7f8-2279-431c-9ae2-81844ba85c56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn"
Jul 30 00:57:49.638834 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.638823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/857ae7f8-2279-431c-9ae2-81844ba85c56-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn\" (UID: \"857ae7f8-2279-431c-9ae2-81844ba85c56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn"
Jul 30 00:57:49.639669 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.639650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/857ae7f8-2279-431c-9ae2-81844ba85c56-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn\" (UID: \"857ae7f8-2279-431c-9ae2-81844ba85c56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn"
Jul 30 00:57:49.646892 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.646864    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqtks\" (UniqueName: \"kubernetes.io/projected/857ae7f8-2279-431c-9ae2-81844ba85c56-kube-api-access-tqtks\") pod \"payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn\" (UID: \"857ae7f8-2279-431c-9ae2-81844ba85c56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn"
Jul 30 00:57:49.740146 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.740118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v8sj\" (UniqueName: \"kubernetes.io/projected/79d43c8e-cde1-45c4-9230-b50f062ec0d7-kube-api-access-5v8sj\") pod \"payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5\" (UID: \"79d43c8e-cde1-45c4-9230-b50f062ec0d7\") " pod="openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5"
Jul 30 00:57:49.740567 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.740153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d43c8e-cde1-45c4-9230-b50f062ec0d7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5\" (UID: \"79d43c8e-cde1-45c4-9230-b50f062ec0d7\") " pod="openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5"
Jul 30 00:57:49.740832 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.740813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d43c8e-cde1-45c4-9230-b50f062ec0d7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5\" (UID: \"79d43c8e-cde1-45c4-9230-b50f062ec0d7\") " pod="openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5"
Jul 30 00:57:49.748475 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.748449    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v8sj\" (UniqueName: \"kubernetes.io/projected/79d43c8e-cde1-45c4-9230-b50f062ec0d7-kube-api-access-5v8sj\") pod \"payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5\" (UID: \"79d43c8e-cde1-45c4-9230-b50f062ec0d7\") " pod="openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5"
Jul 30 00:57:49.819862 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.819791    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn"
Jul 30 00:57:49.825178 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.825156    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"
Jul 30 00:57:49.863159 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.863113    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5"
Jul 30 00:57:49.987179 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:49.987087    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn"]
Jul 30 00:57:49.990122 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:57:49.990070    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857ae7f8_2279_431c_9ae2_81844ba85c56.slice/crio-1ecf3bd6c55c38e3d12babaadfe13a372e0dc38e32e5fecb4a11fce1c546c4ac WatchSource:0}: Error finding container 1ecf3bd6c55c38e3d12babaadfe13a372e0dc38e32e5fecb4a11fce1c546c4ac: Status 404 returned error can't find the container with id 1ecf3bd6c55c38e3d12babaadfe13a372e0dc38e32e5fecb4a11fce1c546c4ac
Jul 30 00:57:50.004863 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:50.004837    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"]
Jul 30 00:57:50.008766 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:57:50.008734    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e10582_b671_4ba8_b4b0_b7d197c60724.slice/crio-e865cfe01f9c2aeac0bc83269d3ca812173cf86d1c652eab122ef41e188bb7de WatchSource:0}: Error finding container e865cfe01f9c2aeac0bc83269d3ca812173cf86d1c652eab122ef41e188bb7de: Status 404 returned error can't find the container with id e865cfe01f9c2aeac0bc83269d3ca812173cf86d1c652eab122ef41e188bb7de
Jul 30 00:57:50.042851 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:50.042806    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5"]
Jul 30 00:57:50.047948 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:57:50.047923    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d43c8e_cde1_45c4_9230_b50f062ec0d7.slice/crio-f00e3b93754b99d7c5e47e5e9773e110c16f307c8e9106210055970d24f4fd14 WatchSource:0}: Error finding container f00e3b93754b99d7c5e47e5e9773e110c16f307c8e9106210055970d24f4fd14: Status 404 returned error can't find the container with id f00e3b93754b99d7c5e47e5e9773e110c16f307c8e9106210055970d24f4fd14
Jul 30 00:57:50.919038 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:50.918999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5" event={"ID":"79d43c8e-cde1-45c4-9230-b50f062ec0d7","Type":"ContainerStarted","Data":"9bff1233c8ba65ff47cf706f58d2ec59704ef848cc621835c7a16dfda94abd30"}
Jul 30 00:57:50.919209 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:50.919058    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5" event={"ID":"79d43c8e-cde1-45c4-9230-b50f062ec0d7","Type":"ContainerStarted","Data":"f00e3b93754b99d7c5e47e5e9773e110c16f307c8e9106210055970d24f4fd14"}
Jul 30 00:57:50.919209 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:50.919116    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5"
Jul 30 00:57:50.920635 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:50.920610    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn" event={"ID":"857ae7f8-2279-431c-9ae2-81844ba85c56","Type":"ContainerStarted","Data":"6df05ce9f2c613ac3ebc1015ae8dcbb67832153880100d3313d4de175acce953"}
Jul 30 00:57:50.920749 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:50.920657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn" event={"ID":"857ae7f8-2279-431c-9ae2-81844ba85c56","Type":"ContainerStarted","Data":"1ecf3bd6c55c38e3d12babaadfe13a372e0dc38e32e5fecb4a11fce1c546c4ac"}
Jul 30 00:57:50.920749 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:50.920679    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn"
Jul 30 00:57:50.922216 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:50.922189    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n" event={"ID":"b9e10582-b671-4ba8-b4b0-b7d197c60724","Type":"ContainerStarted","Data":"e659da51791073eb49ac39ad2672588af6c580c0164c8bd9a2d10fe86b7c374c"}
Jul 30 00:57:50.922324 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:50.922223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n" event={"ID":"b9e10582-b671-4ba8-b4b0-b7d197c60724","Type":"ContainerStarted","Data":"e865cfe01f9c2aeac0bc83269d3ca812173cf86d1c652eab122ef41e188bb7de"}
Jul 30 00:57:50.922324 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:50.922278    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"
Jul 30 00:57:50.936648 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:50.936607    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5" podStartSLOduration=1.936593091 podStartE2EDuration="1.936593091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:57:50.934196062 +0000 UTC m=+2396.231462995" watchObservedRunningTime="2026-07-30 00:57:50.936593091 +0000 UTC m=+2396.233860024"
Jul 30 00:57:50.953421 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:50.953381    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn" podStartSLOduration=1.953367495 podStartE2EDuration="1.953367495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:57:50.950963418 +0000 UTC m=+2396.248230351" watchObservedRunningTime="2026-07-30 00:57:50.953367495 +0000 UTC m=+2396.250634428"
Jul 30 00:57:50.967442 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:50.967400    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n" podStartSLOduration=2.253179568 podStartE2EDuration="2.967386962s" totalImagesPullingTime="714.207394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:57:50.966593248 +0000 UTC m=+2396.263860180" watchObservedRunningTime="2026-07-30 00:57:50.967386962 +0000 UTC m=+2396.264653894"
Jul 30 00:57:55.350021 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:55.349988    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:57:55.353880 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:55.353858    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 00:57:55.570420 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:55.570389    2579 scope.go:117] "RemoveContainer" containerID="d030e563636dc06e1227697e6b517fd1c5081402c0676ea0ecd683090986a19c"
Jul 30 00:57:56.932311 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:57:56.932279    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"
Jul 30 00:58:01.928726 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:01.928696    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn"
Jul 30 00:58:01.929182 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:01.928766    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5"
Jul 30 00:58:30.992803 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:30.992717    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"]
Jul 30 00:58:30.993331 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:30.992965    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp" podUID="64acc2b5-ff32-4881-93be-204787352ded" containerName="maas-api" containerID="cri-o://ba68b00bc669b794205e05d02c308b68223ea2a4f683abbe0d4d81bb50b895ad" gracePeriod=30
Jul 30 00:58:31.063491 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.063460    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt"]
Jul 30 00:58:31.063764 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.063718    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt" podUID="5f9bd2c2-515f-43c6-bee5-804477b13dae" containerName="payload-processing" containerID="cri-o://6c56ce4b80931099e0983b5fb6bbfcc1ddf99b226d3751274b253e8adcbd03de" gracePeriod=30
Jul 30 00:58:31.071949 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.071101    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p"]
Jul 30 00:58:31.071949 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.071555    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p" podUID="f04df206-0038-4f0e-b1c3-a4d6f7302e9e" containerName="payload-pre-processing" containerID="cri-o://101eca073ac245c9bd7c85eb59985fdef05fd7eb122f1229175646abe604499f" gracePeriod=30
Jul 30 00:58:31.128774 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.128741    2579 generic.go:356] "Generic (PLEG): container finished" podID="64acc2b5-ff32-4881-93be-204787352ded" containerID="ba68b00bc669b794205e05d02c308b68223ea2a4f683abbe0d4d81bb50b895ad" exitCode=0
Jul 30 00:58:31.128954 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.128818    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp" event={"ID":"64acc2b5-ff32-4881-93be-204787352ded","Type":"ContainerDied","Data":"ba68b00bc669b794205e05d02c308b68223ea2a4f683abbe0d4d81bb50b895ad"}
Jul 30 00:58:31.335892 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.335870    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"
Jul 30 00:58:31.357778 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.357752    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt"
Jul 30 00:58:31.362391 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.362368    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p"
Jul 30 00:58:31.508837 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.508809    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04df206-0038-4f0e-b1c3-a4d6f7302e9e-plugins-config-volume\") pod \"f04df206-0038-4f0e-b1c3-a4d6f7302e9e\" (UID: \"f04df206-0038-4f0e-b1c3-a4d6f7302e9e\") "
Jul 30 00:58:31.508980 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.508849    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z69x7\" (UniqueName: \"kubernetes.io/projected/64acc2b5-ff32-4881-93be-204787352ded-kube-api-access-z69x7\") pod \"64acc2b5-ff32-4881-93be-204787352ded\" (UID: \"64acc2b5-ff32-4881-93be-204787352ded\") "
Jul 30 00:58:31.508980 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.508892    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9bd2c2-515f-43c6-bee5-804477b13dae-plugins-config-volume\") pod \"5f9bd2c2-515f-43c6-bee5-804477b13dae\" (UID: \"5f9bd2c2-515f-43c6-bee5-804477b13dae\") "
Jul 30 00:58:31.508980 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.508958    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64acc2b5-ff32-4881-93be-204787352ded-maas-api-tls\") pod \"64acc2b5-ff32-4881-93be-204787352ded\" (UID: \"64acc2b5-ff32-4881-93be-204787352ded\") "
Jul 30 00:58:31.509150 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.509001    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmgn8\" (UniqueName: \"kubernetes.io/projected/f04df206-0038-4f0e-b1c3-a4d6f7302e9e-kube-api-access-lmgn8\") pod \"f04df206-0038-4f0e-b1c3-a4d6f7302e9e\" (UID: \"f04df206-0038-4f0e-b1c3-a4d6f7302e9e\") "
Jul 30 00:58:31.509150 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.509036    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64acc2b5-ff32-4881-93be-204787352ded-openshift-service-ca\") pod \"64acc2b5-ff32-4881-93be-204787352ded\" (UID: \"64acc2b5-ff32-4881-93be-204787352ded\") "
Jul 30 00:58:31.509150 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.509076    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sswg\" (UniqueName: \"kubernetes.io/projected/5f9bd2c2-515f-43c6-bee5-804477b13dae-kube-api-access-7sswg\") pod \"5f9bd2c2-515f-43c6-bee5-804477b13dae\" (UID: \"5f9bd2c2-515f-43c6-bee5-804477b13dae\") "
Jul 30 00:58:31.509403 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.509367    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f04df206-0038-4f0e-b1c3-a4d6f7302e9e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f04df206-0038-4f0e-b1c3-a4d6f7302e9e" (UID: "f04df206-0038-4f0e-b1c3-a4d6f7302e9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:58:31.509482 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.509406    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9bd2c2-515f-43c6-bee5-804477b13dae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5f9bd2c2-515f-43c6-bee5-804477b13dae" (UID: "5f9bd2c2-515f-43c6-bee5-804477b13dae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:58:31.509482 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.509443    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64acc2b5-ff32-4881-93be-204787352ded-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "64acc2b5-ff32-4881-93be-204787352ded" (UID: "64acc2b5-ff32-4881-93be-204787352ded"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:58:31.509734 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.509709    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64acc2b5-ff32-4881-93be-204787352ded-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:58:31.509820 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.509733    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04df206-0038-4f0e-b1c3-a4d6f7302e9e-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:58:31.509820 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.509749    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9bd2c2-515f-43c6-bee5-804477b13dae-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:58:31.511277 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.511231    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64acc2b5-ff32-4881-93be-204787352ded-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "64acc2b5-ff32-4881-93be-204787352ded" (UID: "64acc2b5-ff32-4881-93be-204787352ded"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:58:31.511368 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.511278    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64acc2b5-ff32-4881-93be-204787352ded-kube-api-access-z69x7" (OuterVolumeSpecName: "kube-api-access-z69x7") pod "64acc2b5-ff32-4881-93be-204787352ded" (UID: "64acc2b5-ff32-4881-93be-204787352ded"). InnerVolumeSpecName "kube-api-access-z69x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:58:31.511368 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.511309    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04df206-0038-4f0e-b1c3-a4d6f7302e9e-kube-api-access-lmgn8" (OuterVolumeSpecName: "kube-api-access-lmgn8") pod "f04df206-0038-4f0e-b1c3-a4d6f7302e9e" (UID: "f04df206-0038-4f0e-b1c3-a4d6f7302e9e"). InnerVolumeSpecName "kube-api-access-lmgn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:58:31.511368 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.511311    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9bd2c2-515f-43c6-bee5-804477b13dae-kube-api-access-7sswg" (OuterVolumeSpecName: "kube-api-access-7sswg") pod "5f9bd2c2-515f-43c6-bee5-804477b13dae" (UID: "5f9bd2c2-515f-43c6-bee5-804477b13dae"). InnerVolumeSpecName "kube-api-access-7sswg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:58:31.610560 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.610498    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmgn8\" (UniqueName: \"kubernetes.io/projected/f04df206-0038-4f0e-b1c3-a4d6f7302e9e-kube-api-access-lmgn8\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:58:31.610560 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.610520    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sswg\" (UniqueName: \"kubernetes.io/projected/5f9bd2c2-515f-43c6-bee5-804477b13dae-kube-api-access-7sswg\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:58:31.610560 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.610529    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z69x7\" (UniqueName: \"kubernetes.io/projected/64acc2b5-ff32-4881-93be-204787352ded-kube-api-access-z69x7\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:58:31.610560 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:31.610538    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64acc2b5-ff32-4881-93be-204787352ded-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:58:32.134978 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.134949    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"
Jul 30 00:58:32.135432 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.134978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp" event={"ID":"64acc2b5-ff32-4881-93be-204787352ded","Type":"ContainerDied","Data":"de0e20c7da03a2a03bb9a70054a96d40db60317d7ecba6a393db830959879cea"}
Jul 30 00:58:32.135432 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.135045    2579 scope.go:117] "RemoveContainer" containerID="ba68b00bc669b794205e05d02c308b68223ea2a4f683abbe0d4d81bb50b895ad"
Jul 30 00:58:32.136552 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.136518    2579 generic.go:356] "Generic (PLEG): container finished" podID="f04df206-0038-4f0e-b1c3-a4d6f7302e9e" containerID="101eca073ac245c9bd7c85eb59985fdef05fd7eb122f1229175646abe604499f" exitCode=0
Jul 30 00:58:32.136666 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.136551    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p" event={"ID":"f04df206-0038-4f0e-b1c3-a4d6f7302e9e","Type":"ContainerDied","Data":"101eca073ac245c9bd7c85eb59985fdef05fd7eb122f1229175646abe604499f"}
Jul 30 00:58:32.136666 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.136582    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p"
Jul 30 00:58:32.136666 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.136605    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p" event={"ID":"f04df206-0038-4f0e-b1c3-a4d6f7302e9e","Type":"ContainerDied","Data":"05696f4c9546a5b344089c87a73f6870dd92802f95abec7cb3119b55300fc4de"}
Jul 30 00:58:32.138012 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.137991    2579 generic.go:356] "Generic (PLEG): container finished" podID="5f9bd2c2-515f-43c6-bee5-804477b13dae" containerID="6c56ce4b80931099e0983b5fb6bbfcc1ddf99b226d3751274b253e8adcbd03de" exitCode=0
Jul 30 00:58:32.138103 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.138035    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt"
Jul 30 00:58:32.138103 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.138082    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt" event={"ID":"5f9bd2c2-515f-43c6-bee5-804477b13dae","Type":"ContainerDied","Data":"6c56ce4b80931099e0983b5fb6bbfcc1ddf99b226d3751274b253e8adcbd03de"}
Jul 30 00:58:32.138202 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.138121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt" event={"ID":"5f9bd2c2-515f-43c6-bee5-804477b13dae","Type":"ContainerDied","Data":"a7bcd2dfb289b9e4c15935cf1ba3d5bbdc97526671cb1280adb29227cbf266f5"}
Jul 30 00:58:32.145439 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.145419    2579 scope.go:117] "RemoveContainer" containerID="101eca073ac245c9bd7c85eb59985fdef05fd7eb122f1229175646abe604499f"
Jul 30 00:58:32.153981 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.153962    2579 scope.go:117] "RemoveContainer" containerID="101eca073ac245c9bd7c85eb59985fdef05fd7eb122f1229175646abe604499f"
Jul 30 00:58:32.154227 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:58:32.154209    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101eca073ac245c9bd7c85eb59985fdef05fd7eb122f1229175646abe604499f\": container with ID starting with 101eca073ac245c9bd7c85eb59985fdef05fd7eb122f1229175646abe604499f not found: ID does not exist" containerID="101eca073ac245c9bd7c85eb59985fdef05fd7eb122f1229175646abe604499f"
Jul 30 00:58:32.154704 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.154235    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101eca073ac245c9bd7c85eb59985fdef05fd7eb122f1229175646abe604499f"} err="failed to get container status \"101eca073ac245c9bd7c85eb59985fdef05fd7eb122f1229175646abe604499f\": rpc error: code = NotFound desc = could not find container \"101eca073ac245c9bd7c85eb59985fdef05fd7eb122f1229175646abe604499f\": container with ID starting with 101eca073ac245c9bd7c85eb59985fdef05fd7eb122f1229175646abe604499f not found: ID does not exist"
Jul 30 00:58:32.154767 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.154708    2579 scope.go:117] "RemoveContainer" containerID="6c56ce4b80931099e0983b5fb6bbfcc1ddf99b226d3751274b253e8adcbd03de"
Jul 30 00:58:32.162134 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.162116    2579 scope.go:117] "RemoveContainer" containerID="6c56ce4b80931099e0983b5fb6bbfcc1ddf99b226d3751274b253e8adcbd03de"
Jul 30 00:58:32.162391 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:58:32.162372    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c56ce4b80931099e0983b5fb6bbfcc1ddf99b226d3751274b253e8adcbd03de\": container with ID starting with 6c56ce4b80931099e0983b5fb6bbfcc1ddf99b226d3751274b253e8adcbd03de not found: ID does not exist" containerID="6c56ce4b80931099e0983b5fb6bbfcc1ddf99b226d3751274b253e8adcbd03de"
Jul 30 00:58:32.162447 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.162399    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c56ce4b80931099e0983b5fb6bbfcc1ddf99b226d3751274b253e8adcbd03de"} err="failed to get container status \"6c56ce4b80931099e0983b5fb6bbfcc1ddf99b226d3751274b253e8adcbd03de\": rpc error: code = NotFound desc = could not find container \"6c56ce4b80931099e0983b5fb6bbfcc1ddf99b226d3751274b253e8adcbd03de\": container with ID starting with 6c56ce4b80931099e0983b5fb6bbfcc1ddf99b226d3751274b253e8adcbd03de not found: ID does not exist"
Jul 30 00:58:32.168075 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.168052    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt"]
Jul 30 00:58:32.174584 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.174563    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-772c30-584cd5fd8d-whmlt"]
Jul 30 00:58:32.188757 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.188738    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"]
Jul 30 00:58:32.194501 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.194483    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-772c30-787f5f9cdc-8xjwp"]
Jul 30 00:58:32.206841 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.206820    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p"]
Jul 30 00:58:32.208424 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:32.208407    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-772c30-7b49c8f5d8-mb26p"]
Jul 30 00:58:33.322530 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:33.322498    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f9bd2c2-515f-43c6-bee5-804477b13dae" path="/var/lib/kubelet/pods/5f9bd2c2-515f-43c6-bee5-804477b13dae/volumes"
Jul 30 00:58:33.322879 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:33.322865    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64acc2b5-ff32-4881-93be-204787352ded" path="/var/lib/kubelet/pods/64acc2b5-ff32-4881-93be-204787352ded/volumes"
Jul 30 00:58:33.323175 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:33.323161    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f04df206-0038-4f0e-b1c3-a4d6f7302e9e" path="/var/lib/kubelet/pods/f04df206-0038-4f0e-b1c3-a4d6f7302e9e/volumes"
Jul 30 00:58:37.761116 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:37.761078    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-68c25c-vssnv"]
Jul 30 00:58:37.761686 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:37.761666    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f9bd2c2-515f-43c6-bee5-804477b13dae" containerName="payload-processing"
Jul 30 00:58:37.761765 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:37.761688    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f9bd2c2-515f-43c6-bee5-804477b13dae" containerName="payload-processing"
Jul 30 00:58:37.761765 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:37.761704    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64acc2b5-ff32-4881-93be-204787352ded" containerName="maas-api"
Jul 30 00:58:37.761765 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:37.761711    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="64acc2b5-ff32-4881-93be-204787352ded" containerName="maas-api"
Jul 30 00:58:37.761765 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:37.761750    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f04df206-0038-4f0e-b1c3-a4d6f7302e9e" containerName="payload-pre-processing"
Jul 30 00:58:37.761765 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:37.761759    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f04df206-0038-4f0e-b1c3-a4d6f7302e9e" containerName="payload-pre-processing"
Jul 30 00:58:37.762009 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:37.761855    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f04df206-0038-4f0e-b1c3-a4d6f7302e9e" containerName="payload-pre-processing"
Jul 30 00:58:37.762009 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:37.761869    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f9bd2c2-515f-43c6-bee5-804477b13dae" containerName="payload-processing"
Jul 30 00:58:37.762009 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:37.761885    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="64acc2b5-ff32-4881-93be-204787352ded" containerName="maas-api"
Jul 30 00:58:37.766449 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:37.766428    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-68c25c-vssnv"
Jul 30 00:58:37.769360 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:37.769340    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v8q7j\""
Jul 30 00:58:37.771500 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:37.771481    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-68c25c-vssnv"]
Jul 30 00:58:37.862095 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:37.862064    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dafa0b9-42ec-44bd-831c-6adcd38ad2a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-68c25c-vssnv\" (UID: \"3dafa0b9-42ec-44bd-831c-6adcd38ad2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-68c25c-vssnv"
Jul 30 00:58:37.963176 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:37.963147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dafa0b9-42ec-44bd-831c-6adcd38ad2a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-68c25c-vssnv\" (UID: \"3dafa0b9-42ec-44bd-831c-6adcd38ad2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-68c25c-vssnv"
Jul 30 00:58:37.963781 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:37.963762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dafa0b9-42ec-44bd-831c-6adcd38ad2a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-68c25c-vssnv\" (UID: \"3dafa0b9-42ec-44bd-831c-6adcd38ad2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-68c25c-vssnv"
Jul 30 00:58:38.078989 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:38.078921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-68c25c-vssnv"
Jul 30 00:58:38.216858 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:38.216834    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-68c25c-vssnv"]
Jul 30 00:58:38.219021 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:58:38.218993    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dafa0b9_42ec_44bd_831c_6adcd38ad2a5.slice/crio-925dd1925358f7ae7095febaa75b8125c552af7e18145b41ee51808bef2b9173 WatchSource:0}: Error finding container 925dd1925358f7ae7095febaa75b8125c552af7e18145b41ee51808bef2b9173: Status 404 returned error can't find the container with id 925dd1925358f7ae7095febaa75b8125c552af7e18145b41ee51808bef2b9173
Jul 30 00:58:39.181145 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:39.181113    2579 generic.go:356] "Generic (PLEG): container finished" podID="3dafa0b9-42ec-44bd-831c-6adcd38ad2a5" containerID="f53e4c39b8499285b5c7f4bd9413ea8a19ba24a4b2380aade07403916e8a41f6" exitCode=0
Jul 30 00:58:39.181533 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:39.181188    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-68c25c-vssnv" event={"ID":"3dafa0b9-42ec-44bd-831c-6adcd38ad2a5","Type":"ContainerDied","Data":"f53e4c39b8499285b5c7f4bd9413ea8a19ba24a4b2380aade07403916e8a41f6"}
Jul 30 00:58:39.181533 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:39.181235    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-68c25c-vssnv" event={"ID":"3dafa0b9-42ec-44bd-831c-6adcd38ad2a5","Type":"ContainerStarted","Data":"925dd1925358f7ae7095febaa75b8125c552af7e18145b41ee51808bef2b9173"}
Jul 30 00:58:40.319809 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:40.319783    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-68c25c-vssnv"
Jul 30 00:58:40.387426 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:40.385555    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dafa0b9-42ec-44bd-831c-6adcd38ad2a5-maas-api-service-ca\") pod \"3dafa0b9-42ec-44bd-831c-6adcd38ad2a5\" (UID: \"3dafa0b9-42ec-44bd-831c-6adcd38ad2a5\") "
Jul 30 00:58:40.387604 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:40.387544    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dafa0b9-42ec-44bd-831c-6adcd38ad2a5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3dafa0b9-42ec-44bd-831c-6adcd38ad2a5" (UID: "3dafa0b9-42ec-44bd-831c-6adcd38ad2a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:58:40.486930 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:40.486859    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dafa0b9-42ec-44bd-831c-6adcd38ad2a5-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:58:41.192273 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:41.192233    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-68c25c-vssnv"
Jul 30 00:58:41.192273 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:41.192235    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-68c25c-vssnv" event={"ID":"3dafa0b9-42ec-44bd-831c-6adcd38ad2a5","Type":"ContainerDied","Data":"925dd1925358f7ae7095febaa75b8125c552af7e18145b41ee51808bef2b9173"}
Jul 30 00:58:41.192502 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:41.192308    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="925dd1925358f7ae7095febaa75b8125c552af7e18145b41ee51808bef2b9173"
Jul 30 00:58:53.163635 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.163600    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"]
Jul 30 00:58:53.164443 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.164407    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n" podUID="b9e10582-b671-4ba8-b4b0-b7d197c60724" containerName="maas-api" containerID="cri-o://e659da51791073eb49ac39ad2672588af6c580c0164c8bd9a2d10fe86b7c374c" gracePeriod=30
Jul 30 00:58:53.213787 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.213757    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5"]
Jul 30 00:58:53.214220 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.214184    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5" podUID="79d43c8e-cde1-45c4-9230-b50f062ec0d7" containerName="payload-processing" containerID="cri-o://9bff1233c8ba65ff47cf706f58d2ec59704ef848cc621835c7a16dfda94abd30" gracePeriod=30
Jul 30 00:58:53.217146 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.217119    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn"]
Jul 30 00:58:53.217425 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.217378    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn" podUID="857ae7f8-2279-431c-9ae2-81844ba85c56" containerName="payload-pre-processing" containerID="cri-o://6df05ce9f2c613ac3ebc1015ae8dcbb67832153880100d3313d4de175acce953" gracePeriod=30
Jul 30 00:58:53.342476 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.342422    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-68c25c-vssnv"]
Jul 30 00:58:53.346414 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.346386    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-68c25c-vssnv"]
Jul 30 00:58:53.477974 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.477951    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"
Jul 30 00:58:53.515378 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.515352    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5"
Jul 30 00:58:53.520023 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.519998    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn"
Jul 30 00:58:53.598818 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.598791    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2vf9\" (UniqueName: \"kubernetes.io/projected/b9e10582-b671-4ba8-b4b0-b7d197c60724-kube-api-access-w2vf9\") pod \"b9e10582-b671-4ba8-b4b0-b7d197c60724\" (UID: \"b9e10582-b671-4ba8-b4b0-b7d197c60724\") "
Jul 30 00:58:53.598975 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.598826    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e10582-b671-4ba8-b4b0-b7d197c60724-openshift-service-ca\") pod \"b9e10582-b671-4ba8-b4b0-b7d197c60724\" (UID: \"b9e10582-b671-4ba8-b4b0-b7d197c60724\") "
Jul 30 00:58:53.598975 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.598878    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d43c8e-cde1-45c4-9230-b50f062ec0d7-plugins-config-volume\") pod \"79d43c8e-cde1-45c4-9230-b50f062ec0d7\" (UID: \"79d43c8e-cde1-45c4-9230-b50f062ec0d7\") "
Jul 30 00:58:53.599060 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.599038    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e10582-b671-4ba8-b4b0-b7d197c60724-maas-api-tls\") pod \"b9e10582-b671-4ba8-b4b0-b7d197c60724\" (UID: \"b9e10582-b671-4ba8-b4b0-b7d197c60724\") "
Jul 30 00:58:53.599107 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.599074    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/857ae7f8-2279-431c-9ae2-81844ba85c56-plugins-config-volume\") pod \"857ae7f8-2279-431c-9ae2-81844ba85c56\" (UID: \"857ae7f8-2279-431c-9ae2-81844ba85c56\") "
Jul 30 00:58:53.599159 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.599106    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqtks\" (UniqueName: \"kubernetes.io/projected/857ae7f8-2279-431c-9ae2-81844ba85c56-kube-api-access-tqtks\") pod \"857ae7f8-2279-431c-9ae2-81844ba85c56\" (UID: \"857ae7f8-2279-431c-9ae2-81844ba85c56\") "
Jul 30 00:58:53.599159 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.599145    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v8sj\" (UniqueName: \"kubernetes.io/projected/79d43c8e-cde1-45c4-9230-b50f062ec0d7-kube-api-access-5v8sj\") pod \"79d43c8e-cde1-45c4-9230-b50f062ec0d7\" (UID: \"79d43c8e-cde1-45c4-9230-b50f062ec0d7\") "
Jul 30 00:58:53.599306 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.599206    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d43c8e-cde1-45c4-9230-b50f062ec0d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "79d43c8e-cde1-45c4-9230-b50f062ec0d7" (UID: "79d43c8e-cde1-45c4-9230-b50f062ec0d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:58:53.599306 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.599276    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e10582-b671-4ba8-b4b0-b7d197c60724-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b9e10582-b671-4ba8-b4b0-b7d197c60724" (UID: "b9e10582-b671-4ba8-b4b0-b7d197c60724"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:58:53.599585 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.599527    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/857ae7f8-2279-431c-9ae2-81844ba85c56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "857ae7f8-2279-431c-9ae2-81844ba85c56" (UID: "857ae7f8-2279-431c-9ae2-81844ba85c56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:58:53.599585 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.599562    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e10582-b671-4ba8-b4b0-b7d197c60724-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:58:53.599585 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.599578    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d43c8e-cde1-45c4-9230-b50f062ec0d7-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:58:53.601315 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.601276    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857ae7f8-2279-431c-9ae2-81844ba85c56-kube-api-access-tqtks" (OuterVolumeSpecName: "kube-api-access-tqtks") pod "857ae7f8-2279-431c-9ae2-81844ba85c56" (UID: "857ae7f8-2279-431c-9ae2-81844ba85c56"). InnerVolumeSpecName "kube-api-access-tqtks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:58:53.601404 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.601345    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d43c8e-cde1-45c4-9230-b50f062ec0d7-kube-api-access-5v8sj" (OuterVolumeSpecName: "kube-api-access-5v8sj") pod "79d43c8e-cde1-45c4-9230-b50f062ec0d7" (UID: "79d43c8e-cde1-45c4-9230-b50f062ec0d7"). InnerVolumeSpecName "kube-api-access-5v8sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:58:53.601515 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.601497    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e10582-b671-4ba8-b4b0-b7d197c60724-kube-api-access-w2vf9" (OuterVolumeSpecName: "kube-api-access-w2vf9") pod "b9e10582-b671-4ba8-b4b0-b7d197c60724" (UID: "b9e10582-b671-4ba8-b4b0-b7d197c60724"). InnerVolumeSpecName "kube-api-access-w2vf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:58:53.601558 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.601501    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9e10582-b671-4ba8-b4b0-b7d197c60724-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b9e10582-b671-4ba8-b4b0-b7d197c60724" (UID: "b9e10582-b671-4ba8-b4b0-b7d197c60724"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:58:53.700288 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.700225    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2vf9\" (UniqueName: \"kubernetes.io/projected/b9e10582-b671-4ba8-b4b0-b7d197c60724-kube-api-access-w2vf9\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:58:53.700288 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.700269    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e10582-b671-4ba8-b4b0-b7d197c60724-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:58:53.700288 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.700278    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/857ae7f8-2279-431c-9ae2-81844ba85c56-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:58:53.700288 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.700288    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqtks\" (UniqueName: \"kubernetes.io/projected/857ae7f8-2279-431c-9ae2-81844ba85c56-kube-api-access-tqtks\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:58:53.700504 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:53.700297    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v8sj\" (UniqueName: \"kubernetes.io/projected/79d43c8e-cde1-45c4-9230-b50f062ec0d7-kube-api-access-5v8sj\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 00:58:54.260449 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.260418    2579 generic.go:356] "Generic (PLEG): container finished" podID="79d43c8e-cde1-45c4-9230-b50f062ec0d7" containerID="9bff1233c8ba65ff47cf706f58d2ec59704ef848cc621835c7a16dfda94abd30" exitCode=0
Jul 30 00:58:54.260869 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.260460    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5"
Jul 30 00:58:54.260869 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.260502    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5" event={"ID":"79d43c8e-cde1-45c4-9230-b50f062ec0d7","Type":"ContainerDied","Data":"9bff1233c8ba65ff47cf706f58d2ec59704ef848cc621835c7a16dfda94abd30"}
Jul 30 00:58:54.260869 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.260569    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5" event={"ID":"79d43c8e-cde1-45c4-9230-b50f062ec0d7","Type":"ContainerDied","Data":"f00e3b93754b99d7c5e47e5e9773e110c16f307c8e9106210055970d24f4fd14"}
Jul 30 00:58:54.260869 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.260635    2579 scope.go:117] "RemoveContainer" containerID="9bff1233c8ba65ff47cf706f58d2ec59704ef848cc621835c7a16dfda94abd30"
Jul 30 00:58:54.262011 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.261968    2579 generic.go:356] "Generic (PLEG): container finished" podID="857ae7f8-2279-431c-9ae2-81844ba85c56" containerID="6df05ce9f2c613ac3ebc1015ae8dcbb67832153880100d3313d4de175acce953" exitCode=0
Jul 30 00:58:54.262069 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.262015    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn"
Jul 30 00:58:54.262069 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.262057    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn" event={"ID":"857ae7f8-2279-431c-9ae2-81844ba85c56","Type":"ContainerDied","Data":"6df05ce9f2c613ac3ebc1015ae8dcbb67832153880100d3313d4de175acce953"}
Jul 30 00:58:54.262161 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.262117    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn" event={"ID":"857ae7f8-2279-431c-9ae2-81844ba85c56","Type":"ContainerDied","Data":"1ecf3bd6c55c38e3d12babaadfe13a372e0dc38e32e5fecb4a11fce1c546c4ac"}
Jul 30 00:58:54.263605 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.263583    2579 generic.go:356] "Generic (PLEG): container finished" podID="b9e10582-b671-4ba8-b4b0-b7d197c60724" containerID="e659da51791073eb49ac39ad2672588af6c580c0164c8bd9a2d10fe86b7c374c" exitCode=0
Jul 30 00:58:54.263722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.263612    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n" event={"ID":"b9e10582-b671-4ba8-b4b0-b7d197c60724","Type":"ContainerDied","Data":"e659da51791073eb49ac39ad2672588af6c580c0164c8bd9a2d10fe86b7c374c"}
Jul 30 00:58:54.263722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.263643    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"
Jul 30 00:58:54.263722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.263656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n" event={"ID":"b9e10582-b671-4ba8-b4b0-b7d197c60724","Type":"ContainerDied","Data":"e865cfe01f9c2aeac0bc83269d3ca812173cf86d1c652eab122ef41e188bb7de"}
Jul 30 00:58:54.271329 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.271313    2579 scope.go:117] "RemoveContainer" containerID="9bff1233c8ba65ff47cf706f58d2ec59704ef848cc621835c7a16dfda94abd30"
Jul 30 00:58:54.271575 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:58:54.271556    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bff1233c8ba65ff47cf706f58d2ec59704ef848cc621835c7a16dfda94abd30\": container with ID starting with 9bff1233c8ba65ff47cf706f58d2ec59704ef848cc621835c7a16dfda94abd30 not found: ID does not exist" containerID="9bff1233c8ba65ff47cf706f58d2ec59704ef848cc621835c7a16dfda94abd30"
Jul 30 00:58:54.271641 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.271584    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bff1233c8ba65ff47cf706f58d2ec59704ef848cc621835c7a16dfda94abd30"} err="failed to get container status \"9bff1233c8ba65ff47cf706f58d2ec59704ef848cc621835c7a16dfda94abd30\": rpc error: code = NotFound desc = could not find container \"9bff1233c8ba65ff47cf706f58d2ec59704ef848cc621835c7a16dfda94abd30\": container with ID starting with 9bff1233c8ba65ff47cf706f58d2ec59704ef848cc621835c7a16dfda94abd30 not found: ID does not exist"
Jul 30 00:58:54.271641 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.271601    2579 scope.go:117] "RemoveContainer" containerID="6df05ce9f2c613ac3ebc1015ae8dcbb67832153880100d3313d4de175acce953"
Jul 30 00:58:54.279398 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.279381    2579 scope.go:117] "RemoveContainer" containerID="6df05ce9f2c613ac3ebc1015ae8dcbb67832153880100d3313d4de175acce953"
Jul 30 00:58:54.279647 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:58:54.279627    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df05ce9f2c613ac3ebc1015ae8dcbb67832153880100d3313d4de175acce953\": container with ID starting with 6df05ce9f2c613ac3ebc1015ae8dcbb67832153880100d3313d4de175acce953 not found: ID does not exist" containerID="6df05ce9f2c613ac3ebc1015ae8dcbb67832153880100d3313d4de175acce953"
Jul 30 00:58:54.279722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.279658    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df05ce9f2c613ac3ebc1015ae8dcbb67832153880100d3313d4de175acce953"} err="failed to get container status \"6df05ce9f2c613ac3ebc1015ae8dcbb67832153880100d3313d4de175acce953\": rpc error: code = NotFound desc = could not find container \"6df05ce9f2c613ac3ebc1015ae8dcbb67832153880100d3313d4de175acce953\": container with ID starting with 6df05ce9f2c613ac3ebc1015ae8dcbb67832153880100d3313d4de175acce953 not found: ID does not exist"
Jul 30 00:58:54.279722 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.279682    2579 scope.go:117] "RemoveContainer" containerID="e659da51791073eb49ac39ad2672588af6c580c0164c8bd9a2d10fe86b7c374c"
Jul 30 00:58:54.284866 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.284836    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5"]
Jul 30 00:58:54.287557 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.287542    2579 scope.go:117] "RemoveContainer" containerID="e659da51791073eb49ac39ad2672588af6c580c0164c8bd9a2d10fe86b7c374c"
Jul 30 00:58:54.287836 ip-10-0-135-68 kubenswrapper[2579]: E0730 00:58:54.287811    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e659da51791073eb49ac39ad2672588af6c580c0164c8bd9a2d10fe86b7c374c\": container with ID starting with e659da51791073eb49ac39ad2672588af6c580c0164c8bd9a2d10fe86b7c374c not found: ID does not exist" containerID="e659da51791073eb49ac39ad2672588af6c580c0164c8bd9a2d10fe86b7c374c"
Jul 30 00:58:54.287938 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.287842    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e659da51791073eb49ac39ad2672588af6c580c0164c8bd9a2d10fe86b7c374c"} err="failed to get container status \"e659da51791073eb49ac39ad2672588af6c580c0164c8bd9a2d10fe86b7c374c\": rpc error: code = NotFound desc = could not find container \"e659da51791073eb49ac39ad2672588af6c580c0164c8bd9a2d10fe86b7c374c\": container with ID starting with e659da51791073eb49ac39ad2672588af6c580c0164c8bd9a2d10fe86b7c374c not found: ID does not exist"
Jul 30 00:58:54.290105 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.290086    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-68c25c-c49bd77fb-stfr5"]
Jul 30 00:58:54.300687 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.300661    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"]
Jul 30 00:58:54.304483 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.304464    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-68c25c-f955f9969-zsq4n"]
Jul 30 00:58:54.315588 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.315568    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn"]
Jul 30 00:58:54.319663 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:54.319643    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-68c25c-674b657f44-zv4pn"]
Jul 30 00:58:55.322701 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:55.322670    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dafa0b9-42ec-44bd-831c-6adcd38ad2a5" path="/var/lib/kubelet/pods/3dafa0b9-42ec-44bd-831c-6adcd38ad2a5/volumes"
Jul 30 00:58:55.323046 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:55.323018    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79d43c8e-cde1-45c4-9230-b50f062ec0d7" path="/var/lib/kubelet/pods/79d43c8e-cde1-45c4-9230-b50f062ec0d7/volumes"
Jul 30 00:58:55.323347 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:55.323333    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="857ae7f8-2279-431c-9ae2-81844ba85c56" path="/var/lib/kubelet/pods/857ae7f8-2279-431c-9ae2-81844ba85c56/volumes"
Jul 30 00:58:55.323672 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:58:55.323661    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9e10582-b671-4ba8-b4b0-b7d197c60724" path="/var/lib/kubelet/pods/b9e10582-b671-4ba8-b4b0-b7d197c60724/volumes"
Jul 30 00:59:04.282173 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.282136    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"]
Jul 30 00:59:04.282681 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.282605    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79d43c8e-cde1-45c4-9230-b50f062ec0d7" containerName="payload-processing"
Jul 30 00:59:04.282681 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.282620    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79d43c8e-cde1-45c4-9230-b50f062ec0d7" containerName="payload-processing"
Jul 30 00:59:04.282681 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.282633    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="857ae7f8-2279-431c-9ae2-81844ba85c56" containerName="payload-pre-processing"
Jul 30 00:59:04.282681 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.282638    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="857ae7f8-2279-431c-9ae2-81844ba85c56" containerName="payload-pre-processing"
Jul 30 00:59:04.282681 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.282653    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9e10582-b671-4ba8-b4b0-b7d197c60724" containerName="maas-api"
Jul 30 00:59:04.282681 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.282658    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9e10582-b671-4ba8-b4b0-b7d197c60724" containerName="maas-api"
Jul 30 00:59:04.282681 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.282664    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dafa0b9-42ec-44bd-831c-6adcd38ad2a5" containerName="revoke-keys"
Jul 30 00:59:04.282681 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.282669    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dafa0b9-42ec-44bd-831c-6adcd38ad2a5" containerName="revoke-keys"
Jul 30 00:59:04.282976 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.282724    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="79d43c8e-cde1-45c4-9230-b50f062ec0d7" containerName="payload-processing"
Jul 30 00:59:04.282976 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.282732    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="857ae7f8-2279-431c-9ae2-81844ba85c56" containerName="payload-pre-processing"
Jul 30 00:59:04.282976 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.282742    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dafa0b9-42ec-44bd-831c-6adcd38ad2a5" containerName="revoke-keys"
Jul 30 00:59:04.282976 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.282748    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9e10582-b671-4ba8-b4b0-b7d197c60724" containerName="maas-api"
Jul 30 00:59:04.285673 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.285657    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"
Jul 30 00:59:04.288398 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.288376    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-a47fe8-serving-cert\""
Jul 30 00:59:04.295369 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.295342    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"]
Jul 30 00:59:04.396364 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.396334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaeec7a0-65a9-4179-b8e2-9fc07d847f5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q\" (UID: \"eaeec7a0-65a9-4179-b8e2-9fc07d847f5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"
Jul 30 00:59:04.396519 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.396398    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaeec7a0-65a9-4179-b8e2-9fc07d847f5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q\" (UID: \"eaeec7a0-65a9-4179-b8e2-9fc07d847f5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"
Jul 30 00:59:04.396519 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.396466    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbr42\" (UniqueName: \"kubernetes.io/projected/eaeec7a0-65a9-4179-b8e2-9fc07d847f5a-kube-api-access-rbr42\") pod \"maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q\" (UID: \"eaeec7a0-65a9-4179-b8e2-9fc07d847f5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"
Jul 30 00:59:04.497669 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.497640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaeec7a0-65a9-4179-b8e2-9fc07d847f5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q\" (UID: \"eaeec7a0-65a9-4179-b8e2-9fc07d847f5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"
Jul 30 00:59:04.497856 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.497698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaeec7a0-65a9-4179-b8e2-9fc07d847f5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q\" (UID: \"eaeec7a0-65a9-4179-b8e2-9fc07d847f5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"
Jul 30 00:59:04.497902 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.497863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbr42\" (UniqueName: \"kubernetes.io/projected/eaeec7a0-65a9-4179-b8e2-9fc07d847f5a-kube-api-access-rbr42\") pod \"maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q\" (UID: \"eaeec7a0-65a9-4179-b8e2-9fc07d847f5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"
Jul 30 00:59:04.498346 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.498322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaeec7a0-65a9-4179-b8e2-9fc07d847f5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q\" (UID: \"eaeec7a0-65a9-4179-b8e2-9fc07d847f5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"
Jul 30 00:59:04.500103 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.500083    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaeec7a0-65a9-4179-b8e2-9fc07d847f5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q\" (UID: \"eaeec7a0-65a9-4179-b8e2-9fc07d847f5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"
Jul 30 00:59:04.505934 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.505906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbr42\" (UniqueName: \"kubernetes.io/projected/eaeec7a0-65a9-4179-b8e2-9fc07d847f5a-kube-api-access-rbr42\") pod \"maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q\" (UID: \"eaeec7a0-65a9-4179-b8e2-9fc07d847f5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"
Jul 30 00:59:04.596618 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.596552    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"
Jul 30 00:59:04.932588 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.932553    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm"]
Jul 30 00:59:04.937165 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.937107    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm"
Jul 30 00:59:04.939861 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.939839    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"]
Jul 30 00:59:04.940700 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.940646    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-a47fe8-dockercfg-cd9rg\""
Jul 30 00:59:04.940913 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.940897    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-a47fe8\""
Jul 30 00:59:04.945972 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.945952    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm"]
Jul 30 00:59:04.967377 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.967352    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8"]
Jul 30 00:59:04.970706 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.970688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8"
Jul 30 00:59:04.979655 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:04.979622    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8"]
Jul 30 00:59:05.103390 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:05.103361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6x54\" (UniqueName: \"kubernetes.io/projected/1ee3a598-1308-44de-9775-86a0b4bda898-kube-api-access-n6x54\") pod \"payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8\" (UID: \"1ee3a598-1308-44de-9775-86a0b4bda898\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8"
Jul 30 00:59:05.103557 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:05.103431    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsdwq\" (UniqueName: \"kubernetes.io/projected/f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e-kube-api-access-dsdwq\") pod \"payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm\" (UID: \"f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm"
Jul 30 00:59:05.103557 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:05.103465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm\" (UID: \"f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm"
Jul 30 00:59:05.103557 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:05.103537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee3a598-1308-44de-9775-86a0b4bda898-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8\" (UID: \"1ee3a598-1308-44de-9775-86a0b4bda898\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8"
Jul 30 00:59:05.204638 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:05.204562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee3a598-1308-44de-9775-86a0b4bda898-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8\" (UID: \"1ee3a598-1308-44de-9775-86a0b4bda898\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8"
Jul 30 00:59:05.204638 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:05.204608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6x54\" (UniqueName: \"kubernetes.io/projected/1ee3a598-1308-44de-9775-86a0b4bda898-kube-api-access-n6x54\") pod \"payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8\" (UID: \"1ee3a598-1308-44de-9775-86a0b4bda898\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8"
Jul 30 00:59:05.204861 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:05.204656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsdwq\" (UniqueName: \"kubernetes.io/projected/f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e-kube-api-access-dsdwq\") pod \"payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm\" (UID: \"f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm"
Jul 30 00:59:05.204861 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:05.204680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm\" (UID: \"f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm"
Jul 30 00:59:05.205304 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:05.205274    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee3a598-1308-44de-9775-86a0b4bda898-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8\" (UID: \"1ee3a598-1308-44de-9775-86a0b4bda898\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8"
Jul 30 00:59:05.205381 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:05.205327    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm\" (UID: \"f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm"
Jul 30 00:59:05.212628 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:05.212606    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsdwq\" (UniqueName: \"kubernetes.io/projected/f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e-kube-api-access-dsdwq\") pod \"payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm\" (UID: \"f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm"
Jul 30 00:59:05.212733 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:05.212668    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6x54\" (UniqueName: \"kubernetes.io/projected/1ee3a598-1308-44de-9775-86a0b4bda898-kube-api-access-n6x54\") pod \"payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8\" (UID: \"1ee3a598-1308-44de-9775-86a0b4bda898\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8"
Jul 30 00:59:05.264728 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:05.264705    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm"
Jul 30 00:59:05.283819 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:05.283793    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8"
Jul 30 00:59:05.328396 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:05.327289    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q" event={"ID":"eaeec7a0-65a9-4179-b8e2-9fc07d847f5a","Type":"ContainerStarted","Data":"290532e33a954e10fbb1e82a4c13fa0b308cd0953bf94ee36a31034a12ff7a87"}
Jul 30 00:59:05.416001 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:05.415912    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm"]
Jul 30 00:59:05.418780 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:59:05.418755    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fd9bdd_95c0_4ea0_bcdf_c32bfee1f12e.slice/crio-ebbd1c8f1b4bd9197faee17f8301b632e7173e24e380d8fed24e20d6752b537c WatchSource:0}: Error finding container ebbd1c8f1b4bd9197faee17f8301b632e7173e24e380d8fed24e20d6752b537c: Status 404 returned error can't find the container with id ebbd1c8f1b4bd9197faee17f8301b632e7173e24e380d8fed24e20d6752b537c
Jul 30 00:59:05.439840 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:05.439816    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8"]
Jul 30 00:59:05.446446 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:59:05.443832    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee3a598_1308_44de_9775_86a0b4bda898.slice/crio-5d749a99f95651117f6e0e56400afd5f5b90674b3529d6d36c09f0b1fb239c5c WatchSource:0}: Error finding container 5d749a99f95651117f6e0e56400afd5f5b90674b3529d6d36c09f0b1fb239c5c: Status 404 returned error can't find the container with id 5d749a99f95651117f6e0e56400afd5f5b90674b3529d6d36c09f0b1fb239c5c
Jul 30 00:59:06.334868 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:06.334829    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm" event={"ID":"f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e","Type":"ContainerStarted","Data":"7b232827babcbfcb57bdadbf482b220fcfb4116ee59c9377c3370163b4b464b7"}
Jul 30 00:59:06.335387 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:06.334898    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm" event={"ID":"f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e","Type":"ContainerStarted","Data":"ebbd1c8f1b4bd9197faee17f8301b632e7173e24e380d8fed24e20d6752b537c"}
Jul 30 00:59:06.335387 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:06.334956    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm"
Jul 30 00:59:06.336685 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:06.336658    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q" event={"ID":"eaeec7a0-65a9-4179-b8e2-9fc07d847f5a","Type":"ContainerStarted","Data":"509e306a19074348ce7a1051f4ca3e81395ef276da9338bc934258ab6f04d4c5"}
Jul 30 00:59:06.336832 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:06.336709    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"
Jul 30 00:59:06.338423 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:06.338397    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8" event={"ID":"1ee3a598-1308-44de-9775-86a0b4bda898","Type":"ContainerStarted","Data":"dded4ac1c7c056906edafa25e4a900ddbbb923ebb39a145acde5afec6e9f8f6f"}
Jul 30 00:59:06.338544 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:06.338442    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8" event={"ID":"1ee3a598-1308-44de-9775-86a0b4bda898","Type":"ContainerStarted","Data":"5d749a99f95651117f6e0e56400afd5f5b90674b3529d6d36c09f0b1fb239c5c"}
Jul 30 00:59:06.338544 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:06.338474    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8"
Jul 30 00:59:06.350750 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:06.350694    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm" podStartSLOduration=2.350680609 podStartE2EDuration="2.350680609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:59:06.350224695 +0000 UTC m=+2471.647491657" watchObservedRunningTime="2026-07-30 00:59:06.350680609 +0000 UTC m=+2471.647947541"
Jul 30 00:59:06.367392 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:06.367329    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q" podStartSLOduration=1.94471886 podStartE2EDuration="2.367306785s" totalImagesPullingTime="422.587925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:59:06.365476368 +0000 UTC m=+2471.662743301" watchObservedRunningTime="2026-07-30 00:59:06.367306785 +0000 UTC m=+2471.664573709"
Jul 30 00:59:06.389562 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:06.389515    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8" podStartSLOduration=2.389500195 podStartE2EDuration="2.389500195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:59:06.388457574 +0000 UTC m=+2471.685724507" watchObservedRunningTime="2026-07-30 00:59:06.389500195 +0000 UTC m=+2471.686767127"
Jul 30 00:59:12.350162 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:12.350133    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"
Jul 30 00:59:17.346398 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:17.346361    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm"
Jul 30 00:59:17.346977 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:17.346441    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8"
Jul 30 00:59:32.340381 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.340346    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"]
Jul 30 00:59:32.343825 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.343805    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"
Jul 30 00:59:32.347009 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.346981    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-c4c410-serving-cert\""
Jul 30 00:59:32.355679 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.355655    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"]
Jul 30 00:59:32.450050 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.450019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g99x4\" (UniqueName: \"kubernetes.io/projected/8e6fe840-95ea-4f5d-be96-55751b512705-kube-api-access-g99x4\") pod \"maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz\" (UID: \"8e6fe840-95ea-4f5d-be96-55751b512705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"
Jul 30 00:59:32.450203 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.450055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e6fe840-95ea-4f5d-be96-55751b512705-maas-api-tls\") pod \"maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz\" (UID: \"8e6fe840-95ea-4f5d-be96-55751b512705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"
Jul 30 00:59:32.450203 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.450167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6fe840-95ea-4f5d-be96-55751b512705-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz\" (UID: \"8e6fe840-95ea-4f5d-be96-55751b512705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"
Jul 30 00:59:32.551133 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.551094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g99x4\" (UniqueName: \"kubernetes.io/projected/8e6fe840-95ea-4f5d-be96-55751b512705-kube-api-access-g99x4\") pod \"maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz\" (UID: \"8e6fe840-95ea-4f5d-be96-55751b512705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"
Jul 30 00:59:32.551133 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.551135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e6fe840-95ea-4f5d-be96-55751b512705-maas-api-tls\") pod \"maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz\" (UID: \"8e6fe840-95ea-4f5d-be96-55751b512705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"
Jul 30 00:59:32.551403 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.551185    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6fe840-95ea-4f5d-be96-55751b512705-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz\" (UID: \"8e6fe840-95ea-4f5d-be96-55751b512705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"
Jul 30 00:59:32.551794 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.551765    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6fe840-95ea-4f5d-be96-55751b512705-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz\" (UID: \"8e6fe840-95ea-4f5d-be96-55751b512705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"
Jul 30 00:59:32.553632 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.553614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e6fe840-95ea-4f5d-be96-55751b512705-maas-api-tls\") pod \"maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz\" (UID: \"8e6fe840-95ea-4f5d-be96-55751b512705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"
Jul 30 00:59:32.559838 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.559811    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g99x4\" (UniqueName: \"kubernetes.io/projected/8e6fe840-95ea-4f5d-be96-55751b512705-kube-api-access-g99x4\") pod \"maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz\" (UID: \"8e6fe840-95ea-4f5d-be96-55751b512705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"
Jul 30 00:59:32.658264 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.658218    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"
Jul 30 00:59:32.795418 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.795392    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"]
Jul 30 00:59:32.797532 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:59:32.797501    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6fe840_95ea_4f5d_be96_55751b512705.slice/crio-2bae5a9572432a4bbec3b2825da83f073834b16a900b946763c85d4d42143562 WatchSource:0}: Error finding container 2bae5a9572432a4bbec3b2825da83f073834b16a900b946763c85d4d42143562: Status 404 returned error can't find the container with id 2bae5a9572432a4bbec3b2825da83f073834b16a900b946763c85d4d42143562
Jul 30 00:59:32.970351 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.970269    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl"]
Jul 30 00:59:32.974785 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.974769    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl"
Jul 30 00:59:32.984293 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.984267    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-c4c410-dockercfg-tdjgs\""
Jul 30 00:59:32.984496 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.984467    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl"]
Jul 30 00:59:32.984629 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.984610    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-c4c410\""
Jul 30 00:59:32.996828 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:32.996805    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx"]
Jul 30 00:59:33.000284 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.000262    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx"
Jul 30 00:59:33.011211 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.011187    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx"]
Jul 30 00:59:33.056160 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.056131    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbfm8\" (UniqueName: \"kubernetes.io/projected/3f3ee87d-985d-4983-8609-c7feb4e2970d-kube-api-access-xbfm8\") pod \"payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl\" (UID: \"3f3ee87d-985d-4983-8609-c7feb4e2970d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl"
Jul 30 00:59:33.056303 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.056191    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jvkq\" (UniqueName: \"kubernetes.io/projected/637a7f86-1d4e-419a-aea6-bdaff9d4044c-kube-api-access-9jvkq\") pod \"payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx\" (UID: \"637a7f86-1d4e-419a-aea6-bdaff9d4044c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx"
Jul 30 00:59:33.056303 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.056277    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3ee87d-985d-4983-8609-c7feb4e2970d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl\" (UID: \"3f3ee87d-985d-4983-8609-c7feb4e2970d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl"
Jul 30 00:59:33.056415 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.056404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637a7f86-1d4e-419a-aea6-bdaff9d4044c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx\" (UID: \"637a7f86-1d4e-419a-aea6-bdaff9d4044c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx"
Jul 30 00:59:33.157371 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.157338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637a7f86-1d4e-419a-aea6-bdaff9d4044c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx\" (UID: \"637a7f86-1d4e-419a-aea6-bdaff9d4044c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx"
Jul 30 00:59:33.157554 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.157394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbfm8\" (UniqueName: \"kubernetes.io/projected/3f3ee87d-985d-4983-8609-c7feb4e2970d-kube-api-access-xbfm8\") pod \"payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl\" (UID: \"3f3ee87d-985d-4983-8609-c7feb4e2970d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl"
Jul 30 00:59:33.157554 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.157443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jvkq\" (UniqueName: \"kubernetes.io/projected/637a7f86-1d4e-419a-aea6-bdaff9d4044c-kube-api-access-9jvkq\") pod \"payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx\" (UID: \"637a7f86-1d4e-419a-aea6-bdaff9d4044c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx"
Jul 30 00:59:33.157554 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.157480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3ee87d-985d-4983-8609-c7feb4e2970d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl\" (UID: \"3f3ee87d-985d-4983-8609-c7feb4e2970d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl"
Jul 30 00:59:33.158159 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.158133    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637a7f86-1d4e-419a-aea6-bdaff9d4044c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx\" (UID: \"637a7f86-1d4e-419a-aea6-bdaff9d4044c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx"
Jul 30 00:59:33.158275 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.158165    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3ee87d-985d-4983-8609-c7feb4e2970d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl\" (UID: \"3f3ee87d-985d-4983-8609-c7feb4e2970d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl"
Jul 30 00:59:33.165196 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.165171    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jvkq\" (UniqueName: \"kubernetes.io/projected/637a7f86-1d4e-419a-aea6-bdaff9d4044c-kube-api-access-9jvkq\") pod \"payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx\" (UID: \"637a7f86-1d4e-419a-aea6-bdaff9d4044c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx"
Jul 30 00:59:33.165321 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.165229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbfm8\" (UniqueName: \"kubernetes.io/projected/3f3ee87d-985d-4983-8609-c7feb4e2970d-kube-api-access-xbfm8\") pod \"payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl\" (UID: \"3f3ee87d-985d-4983-8609-c7feb4e2970d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl"
Jul 30 00:59:33.287739 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.287720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl"
Jul 30 00:59:33.312229 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.312015    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx"
Jul 30 00:59:33.479012 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.478973    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz" event={"ID":"8e6fe840-95ea-4f5d-be96-55751b512705","Type":"ContainerStarted","Data":"453bcf8e68210224ea6bbff2d4a69f2a8e88f1f877856ee291738f4bb2a635b5"}
Jul 30 00:59:33.479485 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.479038    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz" event={"ID":"8e6fe840-95ea-4f5d-be96-55751b512705","Type":"ContainerStarted","Data":"2bae5a9572432a4bbec3b2825da83f073834b16a900b946763c85d4d42143562"}
Jul 30 00:59:33.479485 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.479126    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"
Jul 30 00:59:33.495787 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.495742    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz" podStartSLOduration=1.033265452 podStartE2EDuration="1.495726444s" totalImagesPullingTime="462.460992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:59:33.494293112 +0000 UTC m=+2498.791560044" watchObservedRunningTime="2026-07-30 00:59:33.495726444 +0000 UTC m=+2498.792993378"
Jul 30 00:59:33.642350 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.642327    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl"]
Jul 30 00:59:33.644152 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:59:33.644128    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3ee87d_985d_4983_8609_c7feb4e2970d.slice/crio-b5661218e8685e1fa1493fb51eb7cfd5f86deca021df47283b8e850c0e243ede WatchSource:0}: Error finding container b5661218e8685e1fa1493fb51eb7cfd5f86deca021df47283b8e850c0e243ede: Status 404 returned error can't find the container with id b5661218e8685e1fa1493fb51eb7cfd5f86deca021df47283b8e850c0e243ede
Jul 30 00:59:33.676790 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:33.676763    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx"]
Jul 30 00:59:33.695454 ip-10-0-135-68 kubenswrapper[2579]: W0730 00:59:33.695427    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637a7f86_1d4e_419a_aea6_bdaff9d4044c.slice/crio-0e22580a0429d031b40796ae37bc1dd0c717e1748af347b210d8b72cee3006b1 WatchSource:0}: Error finding container 0e22580a0429d031b40796ae37bc1dd0c717e1748af347b210d8b72cee3006b1: Status 404 returned error can't find the container with id 0e22580a0429d031b40796ae37bc1dd0c717e1748af347b210d8b72cee3006b1
Jul 30 00:59:34.485552 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:34.485517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx" event={"ID":"637a7f86-1d4e-419a-aea6-bdaff9d4044c","Type":"ContainerStarted","Data":"865abd9a9dd5bbaed47b61908190ae312ec7daedbb924efb7948efd26f06b07e"}
Jul 30 00:59:34.486009 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:34.485578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx" event={"ID":"637a7f86-1d4e-419a-aea6-bdaff9d4044c","Type":"ContainerStarted","Data":"0e22580a0429d031b40796ae37bc1dd0c717e1748af347b210d8b72cee3006b1"}
Jul 30 00:59:34.486009 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:34.485608    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx"
Jul 30 00:59:34.487120 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:34.487096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl" event={"ID":"3f3ee87d-985d-4983-8609-c7feb4e2970d","Type":"ContainerStarted","Data":"568f68876907d6b6dd0989b504df6d38a7204a519cad75f7a94c8a787e251faf"}
Jul 30 00:59:34.487214 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:34.487148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl" event={"ID":"3f3ee87d-985d-4983-8609-c7feb4e2970d","Type":"ContainerStarted","Data":"b5661218e8685e1fa1493fb51eb7cfd5f86deca021df47283b8e850c0e243ede"}
Jul 30 00:59:34.503117 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:34.503066    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx" podStartSLOduration=2.503052489 podStartE2EDuration="2.503052489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:59:34.50063431 +0000 UTC m=+2499.797901242" watchObservedRunningTime="2026-07-30 00:59:34.503052489 +0000 UTC m=+2499.800319421"
Jul 30 00:59:34.516494 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:34.516451    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl" podStartSLOduration=2.516434931 podStartE2EDuration="2.516434931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:59:34.514568423 +0000 UTC m=+2499.811835356" watchObservedRunningTime="2026-07-30 00:59:34.516434931 +0000 UTC m=+2499.813701863"
Jul 30 00:59:35.492266 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:35.492215    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl"
Jul 30 00:59:39.492261 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:39.492223    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"
Jul 30 00:59:45.494774 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:45.494745    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx"
Jul 30 00:59:46.499449 ip-10-0-135-68 kubenswrapper[2579]: I0730 00:59:46.499379    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl"
Jul 30 01:02:55.387759 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:02:55.387684    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 01:02:55.391282 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:02:55.391238    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 01:02:55.669913 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:02:55.669834    2579 scope.go:117] "RemoveContainer" containerID="2ca86533a64aeabd6f58c18f227f4407a169e35b1dd829bc16bcdfdb1987556d"
Jul 30 01:03:55.689745 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:03:55.689715    2579 scope.go:117] "RemoveContainer" containerID="945146d373fefc7e7f623271ad277ec9db85cbbf39d2e8eb59c4fe29300fbbbf"
Jul 30 01:04:55.709677 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:04:55.709645    2579 scope.go:117] "RemoveContainer" containerID="f53e4c39b8499285b5c7f4bd9413ea8a19ba24a4b2380aade07403916e8a41f6"
Jul 30 01:05:59.920662 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:05:59.920585    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"]
Jul 30 01:05:59.924076 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:05:59.924058    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:05:59.931007 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:05:59.930983    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c4c410\"/\"sub-test-model-c4c410-kserve-self-signed-certs\""
Jul 30 01:05:59.931119 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:05:59.930987    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c4c410\"/\"openshift-service-ca.crt\""
Jul 30 01:05:59.931119 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:05:59.930988    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c4c410\"/\"default-dockercfg-62jh9\""
Jul 30 01:05:59.931119 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:05:59.930987    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c4c410\"/\"kube-root-ca.crt\""
Jul 30 01:05:59.950201 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:05:59.950176    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"]
Jul 30 01:06:00.091980 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.091949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkd5g\" (UniqueName: \"kubernetes.io/projected/7c2c1308-92d0-4802-8d94-67d8d2b74333-kube-api-access-xkd5g\") pod \"sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") " pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:00.092163 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.092052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-model-cache\") pod \"sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") " pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:00.092163 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.092090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-tmp-dir\") pod \"sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") " pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:00.092163 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.092143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-home\") pod \"sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") " pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:00.092294 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.092169    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-dshm\") pod \"sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") " pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:00.092294 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.092202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c2c1308-92d0-4802-8d94-67d8d2b74333-tls-certs\") pod \"sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") " pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:00.193070 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.192997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkd5g\" (UniqueName: \"kubernetes.io/projected/7c2c1308-92d0-4802-8d94-67d8d2b74333-kube-api-access-xkd5g\") pod \"sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") " pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:00.193226 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.193068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-model-cache\") pod \"sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") " pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:00.193226 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.193097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-tmp-dir\") pod \"sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") " pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:00.193226 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.193141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-home\") pod \"sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") " pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:00.193226 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.193162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-dshm\") pod \"sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") " pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:00.193226 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.193209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c2c1308-92d0-4802-8d94-67d8d2b74333-tls-certs\") pod \"sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") " pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:00.193564 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.193542    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-tmp-dir\") pod \"sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") " pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:00.193630 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.193577    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-model-cache\") pod \"sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") " pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:00.193630 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.193609    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-home\") pod \"sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") " pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:00.195568 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.195539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-dshm\") pod \"sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") " pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:00.195808 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.195792    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c2c1308-92d0-4802-8d94-67d8d2b74333-tls-certs\") pod \"sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") " pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:00.201414 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.201385    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkd5g\" (UniqueName: \"kubernetes.io/projected/7c2c1308-92d0-4802-8d94-67d8d2b74333-kube-api-access-xkd5g\") pod \"sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") " pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:00.233204 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.233181    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:00.373989 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.373963    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"]
Jul 30 01:06:00.376222 ip-10-0-135-68 kubenswrapper[2579]: W0730 01:06:00.376189    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c2c1308_92d0_4802_8d94_67d8d2b74333.slice/crio-2709a2c09e5a9384ee58c23453467ecba0cd0b1abe53fe69a026580108634295 WatchSource:0}: Error finding container 2709a2c09e5a9384ee58c23453467ecba0cd0b1abe53fe69a026580108634295: Status 404 returned error can't find the container with id 2709a2c09e5a9384ee58c23453467ecba0cd0b1abe53fe69a026580108634295
Jul 30 01:06:00.448890 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.448818    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8" event={"ID":"7c2c1308-92d0-4802-8d94-67d8d2b74333","Type":"ContainerStarted","Data":"ba42bb87260310b467d3728a5033c6d6dc1897a76ff7e35842c391d4908ee546"}
Jul 30 01:06:00.448890 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.448868    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8" event={"ID":"7c2c1308-92d0-4802-8d94-67d8d2b74333","Type":"ContainerStarted","Data":"2709a2c09e5a9384ee58c23453467ecba0cd0b1abe53fe69a026580108634295"}
Jul 30 01:06:00.465838 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:00.465789    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8" podStartSLOduration=1.465773527 podStartE2EDuration="1.465773527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:06:00.464943335 +0000 UTC m=+2885.762210268" watchObservedRunningTime="2026-07-30 01:06:00.465773527 +0000 UTC m=+2885.763040459"
Jul 30 01:06:10.234327 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:10.234292    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:10.234812 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:10.234340    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:10.246262 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:10.246221    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:10.510286 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:10.510185    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:06:57.707219 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:06:57.707180    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"]
Jul 30 01:07:12.727560 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:12.727498    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8" podUID="7c2c1308-92d0-4802-8d94-67d8d2b74333" containerName="main" containerID="cri-o://ba42bb87260310b467d3728a5033c6d6dc1897a76ff7e35842c391d4908ee546" gracePeriod=45
Jul 30 01:07:12.977108 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:12.977085    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:07:13.035198 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.035131    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-home\") pod \"7c2c1308-92d0-4802-8d94-67d8d2b74333\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") "
Jul 30 01:07:13.035198 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.035195    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-tmp-dir\") pod \"7c2c1308-92d0-4802-8d94-67d8d2b74333\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") "
Jul 30 01:07:13.035410 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.035234    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-model-cache\") pod \"7c2c1308-92d0-4802-8d94-67d8d2b74333\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") "
Jul 30 01:07:13.035410 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.035315    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c2c1308-92d0-4802-8d94-67d8d2b74333-tls-certs\") pod \"7c2c1308-92d0-4802-8d94-67d8d2b74333\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") "
Jul 30 01:07:13.035529 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.035422    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkd5g\" (UniqueName: \"kubernetes.io/projected/7c2c1308-92d0-4802-8d94-67d8d2b74333-kube-api-access-xkd5g\") pod \"7c2c1308-92d0-4802-8d94-67d8d2b74333\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") "
Jul 30 01:07:13.035529 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.035452    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-dshm\") pod \"7c2c1308-92d0-4802-8d94-67d8d2b74333\" (UID: \"7c2c1308-92d0-4802-8d94-67d8d2b74333\") "
Jul 30 01:07:13.035529 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.035451    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-home" (OuterVolumeSpecName: "home") pod "7c2c1308-92d0-4802-8d94-67d8d2b74333" (UID: "7c2c1308-92d0-4802-8d94-67d8d2b74333"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 01:07:13.035529 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.035485    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c2c1308-92d0-4802-8d94-67d8d2b74333" (UID: "7c2c1308-92d0-4802-8d94-67d8d2b74333"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 01:07:13.035728 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.035543    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c2c1308-92d0-4802-8d94-67d8d2b74333" (UID: "7c2c1308-92d0-4802-8d94-67d8d2b74333"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 01:07:13.035810 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.035792    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:07:13.035870 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.035816    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:07:13.035870 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.035830    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:07:13.037586 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.037565    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-dshm" (OuterVolumeSpecName: "dshm") pod "7c2c1308-92d0-4802-8d94-67d8d2b74333" (UID: "7c2c1308-92d0-4802-8d94-67d8d2b74333"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 01:07:13.037718 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.037604    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2c1308-92d0-4802-8d94-67d8d2b74333-kube-api-access-xkd5g" (OuterVolumeSpecName: "kube-api-access-xkd5g") pod "7c2c1308-92d0-4802-8d94-67d8d2b74333" (UID: "7c2c1308-92d0-4802-8d94-67d8d2b74333"). InnerVolumeSpecName "kube-api-access-xkd5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 01:07:13.037718 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.037640    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c2c1308-92d0-4802-8d94-67d8d2b74333-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c2c1308-92d0-4802-8d94-67d8d2b74333" (UID: "7c2c1308-92d0-4802-8d94-67d8d2b74333"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 01:07:13.136734 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.136707    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkd5g\" (UniqueName: \"kubernetes.io/projected/7c2c1308-92d0-4802-8d94-67d8d2b74333-kube-api-access-xkd5g\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:07:13.136734 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.136729    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c2c1308-92d0-4802-8d94-67d8d2b74333-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:07:13.136734 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.136739    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c2c1308-92d0-4802-8d94-67d8d2b74333-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:07:13.823006 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.822977    2579 generic.go:356] "Generic (PLEG): container finished" podID="7c2c1308-92d0-4802-8d94-67d8d2b74333" containerID="ba42bb87260310b467d3728a5033c6d6dc1897a76ff7e35842c391d4908ee546" exitCode=0
Jul 30 01:07:13.823425 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.823018    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"
Jul 30 01:07:13.823425 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.823064    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8" event={"ID":"7c2c1308-92d0-4802-8d94-67d8d2b74333","Type":"ContainerDied","Data":"ba42bb87260310b467d3728a5033c6d6dc1897a76ff7e35842c391d4908ee546"}
Jul 30 01:07:13.823425 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.823103    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8" event={"ID":"7c2c1308-92d0-4802-8d94-67d8d2b74333","Type":"ContainerDied","Data":"2709a2c09e5a9384ee58c23453467ecba0cd0b1abe53fe69a026580108634295"}
Jul 30 01:07:13.823425 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.823123    2579 scope.go:117] "RemoveContainer" containerID="ba42bb87260310b467d3728a5033c6d6dc1897a76ff7e35842c391d4908ee546"
Jul 30 01:07:13.831722 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.831704    2579 scope.go:117] "RemoveContainer" containerID="ba42bb87260310b467d3728a5033c6d6dc1897a76ff7e35842c391d4908ee546"
Jul 30 01:07:13.831978 ip-10-0-135-68 kubenswrapper[2579]: E0730 01:07:13.831959    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba42bb87260310b467d3728a5033c6d6dc1897a76ff7e35842c391d4908ee546\": container with ID starting with ba42bb87260310b467d3728a5033c6d6dc1897a76ff7e35842c391d4908ee546 not found: ID does not exist" containerID="ba42bb87260310b467d3728a5033c6d6dc1897a76ff7e35842c391d4908ee546"
Jul 30 01:07:13.832029 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.831987    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba42bb87260310b467d3728a5033c6d6dc1897a76ff7e35842c391d4908ee546"} err="failed to get container status \"ba42bb87260310b467d3728a5033c6d6dc1897a76ff7e35842c391d4908ee546\": rpc error: code = NotFound desc = could not find container \"ba42bb87260310b467d3728a5033c6d6dc1897a76ff7e35842c391d4908ee546\": container with ID starting with ba42bb87260310b467d3728a5033c6d6dc1897a76ff7e35842c391d4908ee546 not found: ID does not exist"
Jul 30 01:07:13.839604 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.839576    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"]
Jul 30 01:07:13.843343 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:13.843318    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c4c410/sub-test-model-c4c410-kserve-79c5f4bb4f-kbxb8"]
Jul 30 01:07:15.322199 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:15.322167    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c2c1308-92d0-4802-8d94-67d8d2b74333" path="/var/lib/kubelet/pods/7c2c1308-92d0-4802-8d94-67d8d2b74333/volumes"
Jul 30 01:07:33.020882 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.020803    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"]
Jul 30 01:07:33.021266 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.021255    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c2c1308-92d0-4802-8d94-67d8d2b74333" containerName="main"
Jul 30 01:07:33.021323 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.021269    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c2c1308-92d0-4802-8d94-67d8d2b74333" containerName="main"
Jul 30 01:07:33.021361 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.021352    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c2c1308-92d0-4802-8d94-67d8d2b74333" containerName="main"
Jul 30 01:07:33.024752 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.024733    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.029508 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.029232    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c4c410\"/\"default-dockercfg-62jh9\""
Jul 30 01:07:33.029508 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.029261    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c4c410\"/\"openshift-service-ca.crt\""
Jul 30 01:07:33.029508 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.029446    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c4c410\"/\"rate-test-model-c4c410-kserve-self-signed-certs\""
Jul 30 01:07:33.029790 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.029523    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c4c410\"/\"kube-root-ca.crt\""
Jul 30 01:07:33.032697 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.032675    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"]
Jul 30 01:07:33.107935 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.107899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-tmp-dir\") pod \"rate-test-model-c4c410-kserve-c46658dc9-k2k88\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") " pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.108101 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.107960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4z4k\" (UniqueName: \"kubernetes.io/projected/91ea0bd2-7907-4078-9fb9-14fa903f4203-kube-api-access-j4z4k\") pod \"rate-test-model-c4c410-kserve-c46658dc9-k2k88\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") " pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.108101 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.108009    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-home\") pod \"rate-test-model-c4c410-kserve-c46658dc9-k2k88\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") " pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.108101 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.108077    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ea0bd2-7907-4078-9fb9-14fa903f4203-tls-certs\") pod \"rate-test-model-c4c410-kserve-c46658dc9-k2k88\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") " pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.108286 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.108131    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-dshm\") pod \"rate-test-model-c4c410-kserve-c46658dc9-k2k88\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") " pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.108286 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.108167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-model-cache\") pod \"rate-test-model-c4c410-kserve-c46658dc9-k2k88\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") " pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.209212 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.209188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ea0bd2-7907-4078-9fb9-14fa903f4203-tls-certs\") pod \"rate-test-model-c4c410-kserve-c46658dc9-k2k88\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") " pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.209479 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.209230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-dshm\") pod \"rate-test-model-c4c410-kserve-c46658dc9-k2k88\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") " pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.209479 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.209366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-model-cache\") pod \"rate-test-model-c4c410-kserve-c46658dc9-k2k88\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") " pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.209479 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.209452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-tmp-dir\") pod \"rate-test-model-c4c410-kserve-c46658dc9-k2k88\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") " pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.209657 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.209507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4z4k\" (UniqueName: \"kubernetes.io/projected/91ea0bd2-7907-4078-9fb9-14fa903f4203-kube-api-access-j4z4k\") pod \"rate-test-model-c4c410-kserve-c46658dc9-k2k88\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") " pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.209657 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.209541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-home\") pod \"rate-test-model-c4c410-kserve-c46658dc9-k2k88\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") " pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.209787 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.209712    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-model-cache\") pod \"rate-test-model-c4c410-kserve-c46658dc9-k2k88\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") " pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.209850 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.209806    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-tmp-dir\") pod \"rate-test-model-c4c410-kserve-c46658dc9-k2k88\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") " pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.209910 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.209891    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-home\") pod \"rate-test-model-c4c410-kserve-c46658dc9-k2k88\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") " pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.211395 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.211377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-dshm\") pod \"rate-test-model-c4c410-kserve-c46658dc9-k2k88\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") " pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.211723 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.211706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ea0bd2-7907-4078-9fb9-14fa903f4203-tls-certs\") pod \"rate-test-model-c4c410-kserve-c46658dc9-k2k88\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") " pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.217135 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.217113    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4z4k\" (UniqueName: \"kubernetes.io/projected/91ea0bd2-7907-4078-9fb9-14fa903f4203-kube-api-access-j4z4k\") pod \"rate-test-model-c4c410-kserve-c46658dc9-k2k88\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") " pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.337210 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.337145    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:33.488306 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.488118    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"]
Jul 30 01:07:33.490603 ip-10-0-135-68 kubenswrapper[2579]: W0730 01:07:33.490570    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91ea0bd2_7907_4078_9fb9_14fa903f4203.slice/crio-57433fb9e63af9d0a6d65f6100fface66332a061567e7b350f085a36bed29194 WatchSource:0}: Error finding container 57433fb9e63af9d0a6d65f6100fface66332a061567e7b350f085a36bed29194: Status 404 returned error can't find the container with id 57433fb9e63af9d0a6d65f6100fface66332a061567e7b350f085a36bed29194
Jul 30 01:07:33.924994 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.924961    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88" event={"ID":"91ea0bd2-7907-4078-9fb9-14fa903f4203","Type":"ContainerStarted","Data":"a0958f79e24d07f6a7446d83b20e2eb022b07aa0cdbe5b4281795170d595f7cf"}
Jul 30 01:07:33.925168 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.925004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88" event={"ID":"91ea0bd2-7907-4078-9fb9-14fa903f4203","Type":"ContainerStarted","Data":"57433fb9e63af9d0a6d65f6100fface66332a061567e7b350f085a36bed29194"}
Jul 30 01:07:33.941731 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:33.941679    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88" podStartSLOduration=1.941662422 podStartE2EDuration="1.941662422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:07:33.941092332 +0000 UTC m=+2979.238359265" watchObservedRunningTime="2026-07-30 01:07:33.941662422 +0000 UTC m=+2979.238929353"
Jul 30 01:07:43.337466 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:43.337438    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:43.337861 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:43.337482    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:43.350451 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:43.350426    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:43.986197 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:43.986162    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:07:55.421294 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:55.421237    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 01:07:55.435707 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:07:55.435677    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 01:08:38.334228 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:38.334189    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"]
Jul 30 01:08:53.352768 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.352682    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88" podUID="91ea0bd2-7907-4078-9fb9-14fa903f4203" containerName="main" containerID="cri-o://a0958f79e24d07f6a7446d83b20e2eb022b07aa0cdbe5b4281795170d595f7cf" gracePeriod=45
Jul 30 01:08:53.604134 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.604078    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:08:53.709722 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.709694    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4z4k\" (UniqueName: \"kubernetes.io/projected/91ea0bd2-7907-4078-9fb9-14fa903f4203-kube-api-access-j4z4k\") pod \"91ea0bd2-7907-4078-9fb9-14fa903f4203\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") "
Jul 30 01:08:53.709905 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.709748    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-model-cache\") pod \"91ea0bd2-7907-4078-9fb9-14fa903f4203\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") "
Jul 30 01:08:53.709905 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.709769    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ea0bd2-7907-4078-9fb9-14fa903f4203-tls-certs\") pod \"91ea0bd2-7907-4078-9fb9-14fa903f4203\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") "
Jul 30 01:08:53.709905 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.709789    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-dshm\") pod \"91ea0bd2-7907-4078-9fb9-14fa903f4203\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") "
Jul 30 01:08:53.709905 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.709855    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-home\") pod \"91ea0bd2-7907-4078-9fb9-14fa903f4203\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") "
Jul 30 01:08:53.709905 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.709870    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-tmp-dir\") pod \"91ea0bd2-7907-4078-9fb9-14fa903f4203\" (UID: \"91ea0bd2-7907-4078-9fb9-14fa903f4203\") "
Jul 30 01:08:53.710178 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.710017    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-model-cache" (OuterVolumeSpecName: "model-cache") pod "91ea0bd2-7907-4078-9fb9-14fa903f4203" (UID: "91ea0bd2-7907-4078-9fb9-14fa903f4203"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 01:08:53.710178 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.710108    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-home" (OuterVolumeSpecName: "home") pod "91ea0bd2-7907-4078-9fb9-14fa903f4203" (UID: "91ea0bd2-7907-4078-9fb9-14fa903f4203"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 01:08:53.710301 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.710173    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "91ea0bd2-7907-4078-9fb9-14fa903f4203" (UID: "91ea0bd2-7907-4078-9fb9-14fa903f4203"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 01:08:53.710301 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.710235    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:08:53.711838 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.711817    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ea0bd2-7907-4078-9fb9-14fa903f4203-kube-api-access-j4z4k" (OuterVolumeSpecName: "kube-api-access-j4z4k") pod "91ea0bd2-7907-4078-9fb9-14fa903f4203" (UID: "91ea0bd2-7907-4078-9fb9-14fa903f4203"). InnerVolumeSpecName "kube-api-access-j4z4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 01:08:53.711931 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.711887    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ea0bd2-7907-4078-9fb9-14fa903f4203-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91ea0bd2-7907-4078-9fb9-14fa903f4203" (UID: "91ea0bd2-7907-4078-9fb9-14fa903f4203"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 01:08:53.711999 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.711979    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-dshm" (OuterVolumeSpecName: "dshm") pod "91ea0bd2-7907-4078-9fb9-14fa903f4203" (UID: "91ea0bd2-7907-4078-9fb9-14fa903f4203"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 01:08:53.811261 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.811224    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4z4k\" (UniqueName: \"kubernetes.io/projected/91ea0bd2-7907-4078-9fb9-14fa903f4203-kube-api-access-j4z4k\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:08:53.811380 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.811264    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ea0bd2-7907-4078-9fb9-14fa903f4203-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:08:53.811380 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.811281    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:08:53.811380 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.811291    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:08:53.811380 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:53.811299    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ea0bd2-7907-4078-9fb9-14fa903f4203-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:08:54.335090 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:54.335061    2579 generic.go:356] "Generic (PLEG): container finished" podID="91ea0bd2-7907-4078-9fb9-14fa903f4203" containerID="a0958f79e24d07f6a7446d83b20e2eb022b07aa0cdbe5b4281795170d595f7cf" exitCode=0
Jul 30 01:08:54.335308 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:54.335103    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"
Jul 30 01:08:54.335308 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:54.335154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88" event={"ID":"91ea0bd2-7907-4078-9fb9-14fa903f4203","Type":"ContainerDied","Data":"a0958f79e24d07f6a7446d83b20e2eb022b07aa0cdbe5b4281795170d595f7cf"}
Jul 30 01:08:54.335308 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:54.335202    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88" event={"ID":"91ea0bd2-7907-4078-9fb9-14fa903f4203","Type":"ContainerDied","Data":"57433fb9e63af9d0a6d65f6100fface66332a061567e7b350f085a36bed29194"}
Jul 30 01:08:54.335308 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:54.335226    2579 scope.go:117] "RemoveContainer" containerID="a0958f79e24d07f6a7446d83b20e2eb022b07aa0cdbe5b4281795170d595f7cf"
Jul 30 01:08:54.346323 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:54.346306    2579 scope.go:117] "RemoveContainer" containerID="a0958f79e24d07f6a7446d83b20e2eb022b07aa0cdbe5b4281795170d595f7cf"
Jul 30 01:08:54.346641 ip-10-0-135-68 kubenswrapper[2579]: E0730 01:08:54.346624    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0958f79e24d07f6a7446d83b20e2eb022b07aa0cdbe5b4281795170d595f7cf\": container with ID starting with a0958f79e24d07f6a7446d83b20e2eb022b07aa0cdbe5b4281795170d595f7cf not found: ID does not exist" containerID="a0958f79e24d07f6a7446d83b20e2eb022b07aa0cdbe5b4281795170d595f7cf"
Jul 30 01:08:54.346711 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:54.346655    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0958f79e24d07f6a7446d83b20e2eb022b07aa0cdbe5b4281795170d595f7cf"} err="failed to get container status \"a0958f79e24d07f6a7446d83b20e2eb022b07aa0cdbe5b4281795170d595f7cf\": rpc error: code = NotFound desc = could not find container \"a0958f79e24d07f6a7446d83b20e2eb022b07aa0cdbe5b4281795170d595f7cf\": container with ID starting with a0958f79e24d07f6a7446d83b20e2eb022b07aa0cdbe5b4281795170d595f7cf not found: ID does not exist"
Jul 30 01:08:54.359061 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:54.359010    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"]
Jul 30 01:08:54.363459 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:54.363431    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c4c410/rate-test-model-c4c410-kserve-c46658dc9-k2k88"]
Jul 30 01:08:55.323640 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:08:55.323610    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91ea0bd2-7907-4078-9fb9-14fa903f4203" path="/var/lib/kubelet/pods/91ea0bd2-7907-4078-9fb9-14fa903f4203/volumes"
Jul 30 01:09:05.741518 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:05.741481    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"]
Jul 30 01:09:05.742138 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:05.742122    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ea0bd2-7907-4078-9fb9-14fa903f4203" containerName="main"
Jul 30 01:09:05.742184 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:05.742141    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ea0bd2-7907-4078-9fb9-14fa903f4203" containerName="main"
Jul 30 01:09:05.742286 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:05.742273    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ea0bd2-7907-4078-9fb9-14fa903f4203" containerName="main"
Jul 30 01:09:05.745739 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:05.745717    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"
Jul 30 01:09:05.748349 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:05.748326    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5912cb-serving-cert\""
Jul 30 01:09:05.754130 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:05.753650    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"]
Jul 30 01:09:05.829094 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:05.829063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bc9a6d-e2e8-4a40-b02e-d27b75af3510-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd\" (UID: \"e8bc9a6d-e2e8-4a40-b02e-d27b75af3510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"
Jul 30 01:09:05.829318 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:05.829104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkgtg\" (UniqueName: \"kubernetes.io/projected/e8bc9a6d-e2e8-4a40-b02e-d27b75af3510-kube-api-access-fkgtg\") pod \"maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd\" (UID: \"e8bc9a6d-e2e8-4a40-b02e-d27b75af3510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"
Jul 30 01:09:05.829318 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:05.829285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8bc9a6d-e2e8-4a40-b02e-d27b75af3510-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd\" (UID: \"e8bc9a6d-e2e8-4a40-b02e-d27b75af3510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"
Jul 30 01:09:05.930843 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:05.930812    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8bc9a6d-e2e8-4a40-b02e-d27b75af3510-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd\" (UID: \"e8bc9a6d-e2e8-4a40-b02e-d27b75af3510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"
Jul 30 01:09:05.931000 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:05.930877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bc9a6d-e2e8-4a40-b02e-d27b75af3510-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd\" (UID: \"e8bc9a6d-e2e8-4a40-b02e-d27b75af3510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"
Jul 30 01:09:05.931000 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:05.930895    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkgtg\" (UniqueName: \"kubernetes.io/projected/e8bc9a6d-e2e8-4a40-b02e-d27b75af3510-kube-api-access-fkgtg\") pod \"maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd\" (UID: \"e8bc9a6d-e2e8-4a40-b02e-d27b75af3510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"
Jul 30 01:09:05.931591 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:05.931566    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bc9a6d-e2e8-4a40-b02e-d27b75af3510-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd\" (UID: \"e8bc9a6d-e2e8-4a40-b02e-d27b75af3510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"
Jul 30 01:09:05.933195 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:05.933174    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8bc9a6d-e2e8-4a40-b02e-d27b75af3510-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd\" (UID: \"e8bc9a6d-e2e8-4a40-b02e-d27b75af3510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"
Jul 30 01:09:05.939736 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:05.939716    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkgtg\" (UniqueName: \"kubernetes.io/projected/e8bc9a6d-e2e8-4a40-b02e-d27b75af3510-kube-api-access-fkgtg\") pod \"maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd\" (UID: \"e8bc9a6d-e2e8-4a40-b02e-d27b75af3510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"
Jul 30 01:09:06.058141 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:06.058064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"
Jul 30 01:09:06.196594 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:06.196565    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"]
Jul 30 01:09:06.198753 ip-10-0-135-68 kubenswrapper[2579]: W0730 01:09:06.198722    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8bc9a6d_e2e8_4a40_b02e_d27b75af3510.slice/crio-4b47403d7822c71ca189948dd83cf70bf253281a7ac875279b3e08ce9734d8be WatchSource:0}: Error finding container 4b47403d7822c71ca189948dd83cf70bf253281a7ac875279b3e08ce9734d8be: Status 404 returned error can't find the container with id 4b47403d7822c71ca189948dd83cf70bf253281a7ac875279b3e08ce9734d8be
Jul 30 01:09:06.200128 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:06.200099    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 01:09:06.366600 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:06.366524    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97"]
Jul 30 01:09:06.371520 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:06.371499    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97"
Jul 30 01:09:06.374347 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:06.374319    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5912cb\""
Jul 30 01:09:06.374347 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:06.374336    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5912cb-dockercfg-r4qkw\""
Jul 30 01:09:06.380299 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:06.380271    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97"]
Jul 30 01:09:06.394731 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:06.394702    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd" event={"ID":"e8bc9a6d-e2e8-4a40-b02e-d27b75af3510","Type":"ContainerStarted","Data":"4b47403d7822c71ca189948dd83cf70bf253281a7ac875279b3e08ce9734d8be"}
Jul 30 01:09:06.537633 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:06.537592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d7d024-fac5-4f01-ac91-36faf1a41cda-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97\" (UID: \"84d7d024-fac5-4f01-ac91-36faf1a41cda\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97"
Jul 30 01:09:06.537834 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:06.537732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stk8s\" (UniqueName: \"kubernetes.io/projected/84d7d024-fac5-4f01-ac91-36faf1a41cda-kube-api-access-stk8s\") pod \"payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97\" (UID: \"84d7d024-fac5-4f01-ac91-36faf1a41cda\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97"
Jul 30 01:09:06.639206 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:06.639175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d7d024-fac5-4f01-ac91-36faf1a41cda-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97\" (UID: \"84d7d024-fac5-4f01-ac91-36faf1a41cda\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97"
Jul 30 01:09:06.639412 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:06.639310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stk8s\" (UniqueName: \"kubernetes.io/projected/84d7d024-fac5-4f01-ac91-36faf1a41cda-kube-api-access-stk8s\") pod \"payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97\" (UID: \"84d7d024-fac5-4f01-ac91-36faf1a41cda\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97"
Jul 30 01:09:06.639924 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:06.639906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d7d024-fac5-4f01-ac91-36faf1a41cda-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97\" (UID: \"84d7d024-fac5-4f01-ac91-36faf1a41cda\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97"
Jul 30 01:09:06.648421 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:06.648400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stk8s\" (UniqueName: \"kubernetes.io/projected/84d7d024-fac5-4f01-ac91-36faf1a41cda-kube-api-access-stk8s\") pod \"payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97\" (UID: \"84d7d024-fac5-4f01-ac91-36faf1a41cda\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97"
Jul 30 01:09:06.684469 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:06.684449    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97"
Jul 30 01:09:06.821430 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:06.821275    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97"]
Jul 30 01:09:06.824156 ip-10-0-135-68 kubenswrapper[2579]: W0730 01:09:06.824131    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d7d024_fac5_4f01_ac91_36faf1a41cda.slice/crio-9c5fd94e517efdc5e83a878ce261c9df90c72f352fa2038ae41c663a424d43be WatchSource:0}: Error finding container 9c5fd94e517efdc5e83a878ce261c9df90c72f352fa2038ae41c663a424d43be: Status 404 returned error can't find the container with id 9c5fd94e517efdc5e83a878ce261c9df90c72f352fa2038ae41c663a424d43be
Jul 30 01:09:07.402046 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:07.402007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd" event={"ID":"e8bc9a6d-e2e8-4a40-b02e-d27b75af3510","Type":"ContainerStarted","Data":"e2e0e9efe40e90f3c1f1cb777e15e9e1cf9aab2863239506f80c2e83eae4d26c"}
Jul 30 01:09:07.402287 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:07.402164    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"
Jul 30 01:09:07.403526 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:07.403503    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97" event={"ID":"84d7d024-fac5-4f01-ac91-36faf1a41cda","Type":"ContainerStarted","Data":"d61082d1c93e2f10102a404e67c62b1a403bdf167f36a62631f40e380dd3a14f"}
Jul 30 01:09:07.403677 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:07.403551    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97" event={"ID":"84d7d024-fac5-4f01-ac91-36faf1a41cda","Type":"ContainerStarted","Data":"9c5fd94e517efdc5e83a878ce261c9df90c72f352fa2038ae41c663a424d43be"}
Jul 30 01:09:07.403677 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:07.403586    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97"
Jul 30 01:09:07.419564 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:07.419516    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd" podStartSLOduration=1.731188629 podStartE2EDuration="2.419503041s" totalImagesPullingTime="688.314412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:09:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:09:07.417011668 +0000 UTC m=+3072.714278600" watchObservedRunningTime="2026-07-30 01:09:07.419503041 +0000 UTC m=+3072.716770001"
Jul 30 01:09:07.431937 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:07.431899    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97" podStartSLOduration=1.43188928 podStartE2EDuration="1.43188928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:09:07.431137636 +0000 UTC m=+3072.728404568" watchObservedRunningTime="2026-07-30 01:09:07.43188928 +0000 UTC m=+3072.729156211"
Jul 30 01:09:13.414512 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:13.414481    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"
Jul 30 01:09:18.411317 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:18.411286    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97"
Jul 30 01:09:35.604807 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:35.604775    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"]
Jul 30 01:09:35.608252 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:35.608221    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"
Jul 30 01:09:35.610809 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:35.610778    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-47d629-serving-cert\""
Jul 30 01:09:35.624473 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:35.624445    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"]
Jul 30 01:09:35.700470 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:35.700442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjr7m\" (UniqueName: \"kubernetes.io/projected/3d582f6c-cdf2-4880-bae4-f3c523131855-kube-api-access-qjr7m\") pod \"maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5\" (UID: \"3d582f6c-cdf2-4880-bae4-f3c523131855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"
Jul 30 01:09:35.700621 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:35.700525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d582f6c-cdf2-4880-bae4-f3c523131855-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5\" (UID: \"3d582f6c-cdf2-4880-bae4-f3c523131855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"
Jul 30 01:09:35.700621 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:35.700559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d582f6c-cdf2-4880-bae4-f3c523131855-maas-api-tls\") pod \"maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5\" (UID: \"3d582f6c-cdf2-4880-bae4-f3c523131855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"
Jul 30 01:09:35.802005 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:35.801978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d582f6c-cdf2-4880-bae4-f3c523131855-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5\" (UID: \"3d582f6c-cdf2-4880-bae4-f3c523131855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"
Jul 30 01:09:35.802193 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:35.802036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d582f6c-cdf2-4880-bae4-f3c523131855-maas-api-tls\") pod \"maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5\" (UID: \"3d582f6c-cdf2-4880-bae4-f3c523131855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"
Jul 30 01:09:35.802193 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:35.802102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjr7m\" (UniqueName: \"kubernetes.io/projected/3d582f6c-cdf2-4880-bae4-f3c523131855-kube-api-access-qjr7m\") pod \"maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5\" (UID: \"3d582f6c-cdf2-4880-bae4-f3c523131855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"
Jul 30 01:09:35.802350 ip-10-0-135-68 kubenswrapper[2579]: E0730 01:09:35.802213    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-serving-cert: secret "maas-api-e2e-ipp-b-47d629-serving-cert" not found
Jul 30 01:09:35.802350 ip-10-0-135-68 kubenswrapper[2579]: E0730 01:09:35.802300    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d582f6c-cdf2-4880-bae4-f3c523131855-maas-api-tls podName:3d582f6c-cdf2-4880-bae4-f3c523131855 nodeName:}" failed. No retries permitted until 2026-07-30 01:09:36.302279774 +0000 UTC m=+3101.599546688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d582f6c-cdf2-4880-bae4-f3c523131855-maas-api-tls") pod "maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5" (UID: "3d582f6c-cdf2-4880-bae4-f3c523131855") : secret "maas-api-e2e-ipp-b-47d629-serving-cert" not found
Jul 30 01:09:35.802721 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:35.802699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d582f6c-cdf2-4880-bae4-f3c523131855-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5\" (UID: \"3d582f6c-cdf2-4880-bae4-f3c523131855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"
Jul 30 01:09:35.812787 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:35.812768    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjr7m\" (UniqueName: \"kubernetes.io/projected/3d582f6c-cdf2-4880-bae4-f3c523131855-kube-api-access-qjr7m\") pod \"maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5\" (UID: \"3d582f6c-cdf2-4880-bae4-f3c523131855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"
Jul 30 01:09:36.307263 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:36.307215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d582f6c-cdf2-4880-bae4-f3c523131855-maas-api-tls\") pod \"maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5\" (UID: \"3d582f6c-cdf2-4880-bae4-f3c523131855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"
Jul 30 01:09:36.309695 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:36.309667    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d582f6c-cdf2-4880-bae4-f3c523131855-maas-api-tls\") pod \"maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5\" (UID: \"3d582f6c-cdf2-4880-bae4-f3c523131855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"
Jul 30 01:09:36.524611 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:36.524581    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"
Jul 30 01:09:36.660218 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:36.660183    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"]
Jul 30 01:09:36.661731 ip-10-0-135-68 kubenswrapper[2579]: W0730 01:09:36.661701    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d582f6c_cdf2_4880_bae4_f3c523131855.slice/crio-33df11b19248e9ad8be10e7582ee9bacb1fcc538b6c7cee26358686a70cddf20 WatchSource:0}: Error finding container 33df11b19248e9ad8be10e7582ee9bacb1fcc538b6c7cee26358686a70cddf20: Status 404 returned error can't find the container with id 33df11b19248e9ad8be10e7582ee9bacb1fcc538b6c7cee26358686a70cddf20
Jul 30 01:09:37.556981 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:37.556937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5" event={"ID":"3d582f6c-cdf2-4880-bae4-f3c523131855","Type":"ContainerStarted","Data":"abca6f1c0f814a95fedc45af1ae2a777176296bd0dbe0fcd145caee2e3f03e7d"}
Jul 30 01:09:37.557141 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:37.557007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5" event={"ID":"3d582f6c-cdf2-4880-bae4-f3c523131855","Type":"ContainerStarted","Data":"33df11b19248e9ad8be10e7582ee9bacb1fcc538b6c7cee26358686a70cddf20"}
Jul 30 01:09:37.557141 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:37.557040    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"
Jul 30 01:09:37.575009 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:37.574959    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5" podStartSLOduration=1.862595295 podStartE2EDuration="2.574943474s" totalImagesPullingTime="712.348179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:09:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:09:37.574128029 +0000 UTC m=+3102.871394961" watchObservedRunningTime="2026-07-30 01:09:37.574943474 +0000 UTC m=+3102.872210454"
Jul 30 01:09:43.568855 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:43.568825    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"
Jul 30 01:09:55.320822 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:09:55.320788    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9k8fv_2e048fda-d493-49d2-955e-a4f1bb2a7f8e/payload-processing/0.log"
Jul 30 01:10:55.475731 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:10:55.475700    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9k8fv_2e048fda-d493-49d2-955e-a4f1bb2a7f8e/payload-processing/0.log"
Jul 30 01:11:03.013611 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.013579    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"]
Jul 30 01:11:03.017560 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.017537    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"
Jul 30 01:11:03.019999 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.019979    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-ff67f5-serving-cert\""
Jul 30 01:11:03.025025 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.025002    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"]
Jul 30 01:11:03.141103 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.141072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60edf95f-501c-4cb3-af18-95d52c548bd9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk\" (UID: \"60edf95f-501c-4cb3-af18-95d52c548bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"
Jul 30 01:11:03.141103 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.141107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60edf95f-501c-4cb3-af18-95d52c548bd9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk\" (UID: \"60edf95f-501c-4cb3-af18-95d52c548bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"
Jul 30 01:11:03.141331 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.141144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdk48\" (UniqueName: \"kubernetes.io/projected/60edf95f-501c-4cb3-af18-95d52c548bd9-kube-api-access-sdk48\") pod \"maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk\" (UID: \"60edf95f-501c-4cb3-af18-95d52c548bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"
Jul 30 01:11:03.242138 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.242097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdk48\" (UniqueName: \"kubernetes.io/projected/60edf95f-501c-4cb3-af18-95d52c548bd9-kube-api-access-sdk48\") pod \"maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk\" (UID: \"60edf95f-501c-4cb3-af18-95d52c548bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"
Jul 30 01:11:03.242322 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.242232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60edf95f-501c-4cb3-af18-95d52c548bd9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk\" (UID: \"60edf95f-501c-4cb3-af18-95d52c548bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"
Jul 30 01:11:03.242322 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.242290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60edf95f-501c-4cb3-af18-95d52c548bd9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk\" (UID: \"60edf95f-501c-4cb3-af18-95d52c548bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"
Jul 30 01:11:03.242973 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.242948    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60edf95f-501c-4cb3-af18-95d52c548bd9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk\" (UID: \"60edf95f-501c-4cb3-af18-95d52c548bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"
Jul 30 01:11:03.244601 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.244581    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60edf95f-501c-4cb3-af18-95d52c548bd9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk\" (UID: \"60edf95f-501c-4cb3-af18-95d52c548bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"
Jul 30 01:11:03.250710 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.250688    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdk48\" (UniqueName: \"kubernetes.io/projected/60edf95f-501c-4cb3-af18-95d52c548bd9-kube-api-access-sdk48\") pod \"maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk\" (UID: \"60edf95f-501c-4cb3-af18-95d52c548bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"
Jul 30 01:11:03.330625 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.330548    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"
Jul 30 01:11:03.472498 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.472471    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"]
Jul 30 01:11:03.475765 ip-10-0-135-68 kubenswrapper[2579]: W0730 01:11:03.475732    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60edf95f_501c_4cb3_af18_95d52c548bd9.slice/crio-11516960d37977eeb92f07becdf869d3fa268b7197859e78e6fce7ff4babe895 WatchSource:0}: Error finding container 11516960d37977eeb92f07becdf869d3fa268b7197859e78e6fce7ff4babe895: Status 404 returned error can't find the container with id 11516960d37977eeb92f07becdf869d3fa268b7197859e78e6fce7ff4babe895
Jul 30 01:11:03.614844 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.614764    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4"]
Jul 30 01:11:03.619857 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.619836    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4"
Jul 30 01:11:03.622966 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.622943    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-ff67f5-dockercfg-v9zzc\""
Jul 30 01:11:03.623292 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.623276    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-ff67f5\""
Jul 30 01:11:03.628875 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.628844    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4"]
Jul 30 01:11:03.640598 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.640574    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd"]
Jul 30 01:11:03.644430 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.644410    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd"
Jul 30 01:11:03.664325 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.664301    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd"]
Jul 30 01:11:03.747160 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.747121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17c51729-1ae6-446c-9008-1054653c7b2f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd\" (UID: \"17c51729-1ae6-446c-9008-1054653c7b2f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd"
Jul 30 01:11:03.747327 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.747186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qgx8\" (UniqueName: \"kubernetes.io/projected/17c51729-1ae6-446c-9008-1054653c7b2f-kube-api-access-2qgx8\") pod \"payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd\" (UID: \"17c51729-1ae6-446c-9008-1054653c7b2f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd"
Jul 30 01:11:03.747327 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.747306    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khb2x\" (UniqueName: \"kubernetes.io/projected/bacf50c0-c60f-4404-a46e-109d921e1acb-kube-api-access-khb2x\") pod \"payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4\" (UID: \"bacf50c0-c60f-4404-a46e-109d921e1acb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4"
Jul 30 01:11:03.747415 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.747390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacf50c0-c60f-4404-a46e-109d921e1acb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4\" (UID: \"bacf50c0-c60f-4404-a46e-109d921e1acb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4"
Jul 30 01:11:03.848885 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.848848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacf50c0-c60f-4404-a46e-109d921e1acb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4\" (UID: \"bacf50c0-c60f-4404-a46e-109d921e1acb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4"
Jul 30 01:11:03.849035 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.848901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17c51729-1ae6-446c-9008-1054653c7b2f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd\" (UID: \"17c51729-1ae6-446c-9008-1054653c7b2f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd"
Jul 30 01:11:03.849035 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.848945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qgx8\" (UniqueName: \"kubernetes.io/projected/17c51729-1ae6-446c-9008-1054653c7b2f-kube-api-access-2qgx8\") pod \"payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd\" (UID: \"17c51729-1ae6-446c-9008-1054653c7b2f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd"
Jul 30 01:11:03.849035 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.849022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khb2x\" (UniqueName: \"kubernetes.io/projected/bacf50c0-c60f-4404-a46e-109d921e1acb-kube-api-access-khb2x\") pod \"payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4\" (UID: \"bacf50c0-c60f-4404-a46e-109d921e1acb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4"
Jul 30 01:11:03.849611 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.849590    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacf50c0-c60f-4404-a46e-109d921e1acb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4\" (UID: \"bacf50c0-c60f-4404-a46e-109d921e1acb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4"
Jul 30 01:11:03.849671 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.849653    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17c51729-1ae6-446c-9008-1054653c7b2f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd\" (UID: \"17c51729-1ae6-446c-9008-1054653c7b2f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd"
Jul 30 01:11:03.859811 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.859784    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khb2x\" (UniqueName: \"kubernetes.io/projected/bacf50c0-c60f-4404-a46e-109d921e1acb-kube-api-access-khb2x\") pod \"payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4\" (UID: \"bacf50c0-c60f-4404-a46e-109d921e1acb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4"
Jul 30 01:11:03.859910 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.859820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qgx8\" (UniqueName: \"kubernetes.io/projected/17c51729-1ae6-446c-9008-1054653c7b2f-kube-api-access-2qgx8\") pod \"payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd\" (UID: \"17c51729-1ae6-446c-9008-1054653c7b2f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd"
Jul 30 01:11:03.933016 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.932887    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4"
Jul 30 01:11:03.954294 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:03.954273    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd"
Jul 30 01:11:04.024093 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:04.024024    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk" event={"ID":"60edf95f-501c-4cb3-af18-95d52c548bd9","Type":"ContainerStarted","Data":"db394a19492656951fd9d588cc4644e54f6508152abd7000dd420a86f65265ea"}
Jul 30 01:11:04.024093 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:04.024086    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk" event={"ID":"60edf95f-501c-4cb3-af18-95d52c548bd9","Type":"ContainerStarted","Data":"11516960d37977eeb92f07becdf869d3fa268b7197859e78e6fce7ff4babe895"}
Jul 30 01:11:04.024662 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:04.024191    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"
Jul 30 01:11:04.048015 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:04.047965    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk" podStartSLOduration=1.590108541 podStartE2EDuration="2.047945017s" totalImagesPullingTime="457.836476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:11:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:11:04.045808934 +0000 UTC m=+3189.343075888" watchObservedRunningTime="2026-07-30 01:11:04.047945017 +0000 UTC m=+3189.345211955"
Jul 30 01:11:04.296176 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:04.296151    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4"]
Jul 30 01:11:04.298906 ip-10-0-135-68 kubenswrapper[2579]: W0730 01:11:04.298879    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbacf50c0_c60f_4404_a46e_109d921e1acb.slice/crio-c05dd067d6b63b889b46834119f6185b7bfc779132e4254cf5f1243ae8ba9484 WatchSource:0}: Error finding container c05dd067d6b63b889b46834119f6185b7bfc779132e4254cf5f1243ae8ba9484: Status 404 returned error can't find the container with id c05dd067d6b63b889b46834119f6185b7bfc779132e4254cf5f1243ae8ba9484
Jul 30 01:11:04.332710 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:04.332685    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd"]
Jul 30 01:11:04.336395 ip-10-0-135-68 kubenswrapper[2579]: W0730 01:11:04.336363    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c51729_1ae6_446c_9008_1054653c7b2f.slice/crio-8ba1c94a530416d0da663a38b13a40812b8a2496b5cd6ea3f879d4950470859b WatchSource:0}: Error finding container 8ba1c94a530416d0da663a38b13a40812b8a2496b5cd6ea3f879d4950470859b: Status 404 returned error can't find the container with id 8ba1c94a530416d0da663a38b13a40812b8a2496b5cd6ea3f879d4950470859b
Jul 30 01:11:05.031197 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:05.031156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4" event={"ID":"bacf50c0-c60f-4404-a46e-109d921e1acb","Type":"ContainerStarted","Data":"08a75fd962cd958f5b3d702743a520a4a08035ebaa1396a5ec5e35b043d43998"}
Jul 30 01:11:05.031655 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:05.031227    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4"
Jul 30 01:11:05.031655 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:05.031266    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4" event={"ID":"bacf50c0-c60f-4404-a46e-109d921e1acb","Type":"ContainerStarted","Data":"c05dd067d6b63b889b46834119f6185b7bfc779132e4254cf5f1243ae8ba9484"}
Jul 30 01:11:05.033073 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:05.033045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd" event={"ID":"17c51729-1ae6-446c-9008-1054653c7b2f","Type":"ContainerStarted","Data":"5c02ebddb1b5cdf9438461ffb32268eb633e3b2b588fdcce8af7c79df56f8348"}
Jul 30 01:11:05.033213 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:05.033096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd" event={"ID":"17c51729-1ae6-446c-9008-1054653c7b2f","Type":"ContainerStarted","Data":"8ba1c94a530416d0da663a38b13a40812b8a2496b5cd6ea3f879d4950470859b"}
Jul 30 01:11:05.033399 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:05.033375    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd"
Jul 30 01:11:05.047797 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:05.047739    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4" podStartSLOduration=2.047721286 podStartE2EDuration="2.047721286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:11:05.047380828 +0000 UTC m=+3190.344647762" watchObservedRunningTime="2026-07-30 01:11:05.047721286 +0000 UTC m=+3190.344988220"
Jul 30 01:11:05.064118 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:05.064074    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd" podStartSLOduration=2.064060293 podStartE2EDuration="2.064060293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:11:05.062614452 +0000 UTC m=+3190.359881385" watchObservedRunningTime="2026-07-30 01:11:05.064060293 +0000 UTC m=+3190.361327224"
Jul 30 01:11:10.039026 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:10.038998    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"
Jul 30 01:11:16.041977 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:16.041941    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd"
Jul 30 01:11:16.042412 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:16.042018    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4"
Jul 30 01:11:27.373358 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.373319    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"]
Jul 30 01:11:27.373896 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.373511    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk" podUID="60edf95f-501c-4cb3-af18-95d52c548bd9" containerName="maas-api" containerID="cri-o://db394a19492656951fd9d588cc4644e54f6508152abd7000dd420a86f65265ea" gracePeriod=30
Jul 30 01:11:27.422844 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.422806    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd"]
Jul 30 01:11:27.423019 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.422997    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd" podUID="17c51729-1ae6-446c-9008-1054653c7b2f" containerName="payload-processing" containerID="cri-o://5c02ebddb1b5cdf9438461ffb32268eb633e3b2b588fdcce8af7c79df56f8348" gracePeriod=30
Jul 30 01:11:27.429210 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.429180    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4"]
Jul 30 01:11:27.429471 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.429427    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4" podUID="bacf50c0-c60f-4404-a46e-109d921e1acb" containerName="payload-pre-processing" containerID="cri-o://08a75fd962cd958f5b3d702743a520a4a08035ebaa1396a5ec5e35b043d43998" gracePeriod=30
Jul 30 01:11:27.738935 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.738911    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd"
Jul 30 01:11:27.742402 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.742382    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"
Jul 30 01:11:27.745916 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.745896    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4"
Jul 30 01:11:27.777584 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.777562    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qgx8\" (UniqueName: \"kubernetes.io/projected/17c51729-1ae6-446c-9008-1054653c7b2f-kube-api-access-2qgx8\") pod \"17c51729-1ae6-446c-9008-1054653c7b2f\" (UID: \"17c51729-1ae6-446c-9008-1054653c7b2f\") "
Jul 30 01:11:27.777743 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.777603    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60edf95f-501c-4cb3-af18-95d52c548bd9-maas-api-tls\") pod \"60edf95f-501c-4cb3-af18-95d52c548bd9\" (UID: \"60edf95f-501c-4cb3-af18-95d52c548bd9\") "
Jul 30 01:11:27.777743 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.777631    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60edf95f-501c-4cb3-af18-95d52c548bd9-openshift-service-ca\") pod \"60edf95f-501c-4cb3-af18-95d52c548bd9\" (UID: \"60edf95f-501c-4cb3-af18-95d52c548bd9\") "
Jul 30 01:11:27.777743 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.777660    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17c51729-1ae6-446c-9008-1054653c7b2f-plugins-config-volume\") pod \"17c51729-1ae6-446c-9008-1054653c7b2f\" (UID: \"17c51729-1ae6-446c-9008-1054653c7b2f\") "
Jul 30 01:11:27.777743 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.777717    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacf50c0-c60f-4404-a46e-109d921e1acb-plugins-config-volume\") pod \"bacf50c0-c60f-4404-a46e-109d921e1acb\" (UID: \"bacf50c0-c60f-4404-a46e-109d921e1acb\") "
Jul 30 01:11:27.777958 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.777798    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khb2x\" (UniqueName: \"kubernetes.io/projected/bacf50c0-c60f-4404-a46e-109d921e1acb-kube-api-access-khb2x\") pod \"bacf50c0-c60f-4404-a46e-109d921e1acb\" (UID: \"bacf50c0-c60f-4404-a46e-109d921e1acb\") "
Jul 30 01:11:27.777958 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.777831    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdk48\" (UniqueName: \"kubernetes.io/projected/60edf95f-501c-4cb3-af18-95d52c548bd9-kube-api-access-sdk48\") pod \"60edf95f-501c-4cb3-af18-95d52c548bd9\" (UID: \"60edf95f-501c-4cb3-af18-95d52c548bd9\") "
Jul 30 01:11:27.778511 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.778458    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bacf50c0-c60f-4404-a46e-109d921e1acb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bacf50c0-c60f-4404-a46e-109d921e1acb" (UID: "bacf50c0-c60f-4404-a46e-109d921e1acb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 01:11:27.779007 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.778867    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c51729-1ae6-446c-9008-1054653c7b2f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "17c51729-1ae6-446c-9008-1054653c7b2f" (UID: "17c51729-1ae6-446c-9008-1054653c7b2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 01:11:27.779193 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.779168    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60edf95f-501c-4cb3-af18-95d52c548bd9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "60edf95f-501c-4cb3-af18-95d52c548bd9" (UID: "60edf95f-501c-4cb3-af18-95d52c548bd9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 01:11:27.780661 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.780637    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60edf95f-501c-4cb3-af18-95d52c548bd9-kube-api-access-sdk48" (OuterVolumeSpecName: "kube-api-access-sdk48") pod "60edf95f-501c-4cb3-af18-95d52c548bd9" (UID: "60edf95f-501c-4cb3-af18-95d52c548bd9"). InnerVolumeSpecName "kube-api-access-sdk48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 01:11:27.780761 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.780702    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bacf50c0-c60f-4404-a46e-109d921e1acb-kube-api-access-khb2x" (OuterVolumeSpecName: "kube-api-access-khb2x") pod "bacf50c0-c60f-4404-a46e-109d921e1acb" (UID: "bacf50c0-c60f-4404-a46e-109d921e1acb"). InnerVolumeSpecName "kube-api-access-khb2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 01:11:27.780828 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.780806    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c51729-1ae6-446c-9008-1054653c7b2f-kube-api-access-2qgx8" (OuterVolumeSpecName: "kube-api-access-2qgx8") pod "17c51729-1ae6-446c-9008-1054653c7b2f" (UID: "17c51729-1ae6-446c-9008-1054653c7b2f"). InnerVolumeSpecName "kube-api-access-2qgx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 01:11:27.781098 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.781079    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60edf95f-501c-4cb3-af18-95d52c548bd9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "60edf95f-501c-4cb3-af18-95d52c548bd9" (UID: "60edf95f-501c-4cb3-af18-95d52c548bd9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 01:11:27.879584 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.879556    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacf50c0-c60f-4404-a46e-109d921e1acb-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:11:27.879584 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.879580    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khb2x\" (UniqueName: \"kubernetes.io/projected/bacf50c0-c60f-4404-a46e-109d921e1acb-kube-api-access-khb2x\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:11:27.879584 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.879591    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdk48\" (UniqueName: \"kubernetes.io/projected/60edf95f-501c-4cb3-af18-95d52c548bd9-kube-api-access-sdk48\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:11:27.879786 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.879600    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qgx8\" (UniqueName: \"kubernetes.io/projected/17c51729-1ae6-446c-9008-1054653c7b2f-kube-api-access-2qgx8\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:11:27.879786 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.879611    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60edf95f-501c-4cb3-af18-95d52c548bd9-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:11:27.879786 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.879619    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60edf95f-501c-4cb3-af18-95d52c548bd9-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:11:27.879786 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:27.879628    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17c51729-1ae6-446c-9008-1054653c7b2f-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:11:28.158491 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.158457    2579 generic.go:356] "Generic (PLEG): container finished" podID="17c51729-1ae6-446c-9008-1054653c7b2f" containerID="5c02ebddb1b5cdf9438461ffb32268eb633e3b2b588fdcce8af7c79df56f8348" exitCode=0
Jul 30 01:11:28.158684 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.158507    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd"
Jul 30 01:11:28.158684 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.158551    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd" event={"ID":"17c51729-1ae6-446c-9008-1054653c7b2f","Type":"ContainerDied","Data":"5c02ebddb1b5cdf9438461ffb32268eb633e3b2b588fdcce8af7c79df56f8348"}
Jul 30 01:11:28.158684 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.158609    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd" event={"ID":"17c51729-1ae6-446c-9008-1054653c7b2f","Type":"ContainerDied","Data":"8ba1c94a530416d0da663a38b13a40812b8a2496b5cd6ea3f879d4950470859b"}
Jul 30 01:11:28.158684 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.158641    2579 scope.go:117] "RemoveContainer" containerID="5c02ebddb1b5cdf9438461ffb32268eb633e3b2b588fdcce8af7c79df56f8348"
Jul 30 01:11:28.160381 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.160357    2579 generic.go:356] "Generic (PLEG): container finished" podID="bacf50c0-c60f-4404-a46e-109d921e1acb" containerID="08a75fd962cd958f5b3d702743a520a4a08035ebaa1396a5ec5e35b043d43998" exitCode=0
Jul 30 01:11:28.160521 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.160396    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4"
Jul 30 01:11:28.160521 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.160468    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4" event={"ID":"bacf50c0-c60f-4404-a46e-109d921e1acb","Type":"ContainerDied","Data":"08a75fd962cd958f5b3d702743a520a4a08035ebaa1396a5ec5e35b043d43998"}
Jul 30 01:11:28.160521 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.160520    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4" event={"ID":"bacf50c0-c60f-4404-a46e-109d921e1acb","Type":"ContainerDied","Data":"c05dd067d6b63b889b46834119f6185b7bfc779132e4254cf5f1243ae8ba9484"}
Jul 30 01:11:28.162291 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.162228    2579 generic.go:356] "Generic (PLEG): container finished" podID="60edf95f-501c-4cb3-af18-95d52c548bd9" containerID="db394a19492656951fd9d588cc4644e54f6508152abd7000dd420a86f65265ea" exitCode=0
Jul 30 01:11:28.162439 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.162317    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"
Jul 30 01:11:28.162542 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.162334    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk" event={"ID":"60edf95f-501c-4cb3-af18-95d52c548bd9","Type":"ContainerDied","Data":"db394a19492656951fd9d588cc4644e54f6508152abd7000dd420a86f65265ea"}
Jul 30 01:11:28.162714 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.162572    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk" event={"ID":"60edf95f-501c-4cb3-af18-95d52c548bd9","Type":"ContainerDied","Data":"11516960d37977eeb92f07becdf869d3fa268b7197859e78e6fce7ff4babe895"}
Jul 30 01:11:28.171284 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.171265    2579 scope.go:117] "RemoveContainer" containerID="5c02ebddb1b5cdf9438461ffb32268eb633e3b2b588fdcce8af7c79df56f8348"
Jul 30 01:11:28.171573 ip-10-0-135-68 kubenswrapper[2579]: E0730 01:11:28.171548    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c02ebddb1b5cdf9438461ffb32268eb633e3b2b588fdcce8af7c79df56f8348\": container with ID starting with 5c02ebddb1b5cdf9438461ffb32268eb633e3b2b588fdcce8af7c79df56f8348 not found: ID does not exist" containerID="5c02ebddb1b5cdf9438461ffb32268eb633e3b2b588fdcce8af7c79df56f8348"
Jul 30 01:11:28.171671 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.171611    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c02ebddb1b5cdf9438461ffb32268eb633e3b2b588fdcce8af7c79df56f8348"} err="failed to get container status \"5c02ebddb1b5cdf9438461ffb32268eb633e3b2b588fdcce8af7c79df56f8348\": rpc error: code = NotFound desc = could not find container \"5c02ebddb1b5cdf9438461ffb32268eb633e3b2b588fdcce8af7c79df56f8348\": container with ID starting with 5c02ebddb1b5cdf9438461ffb32268eb633e3b2b588fdcce8af7c79df56f8348 not found: ID does not exist"
Jul 30 01:11:28.171671 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.171639    2579 scope.go:117] "RemoveContainer" containerID="08a75fd962cd958f5b3d702743a520a4a08035ebaa1396a5ec5e35b043d43998"
Jul 30 01:11:28.180005 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.179988    2579 scope.go:117] "RemoveContainer" containerID="08a75fd962cd958f5b3d702743a520a4a08035ebaa1396a5ec5e35b043d43998"
Jul 30 01:11:28.180341 ip-10-0-135-68 kubenswrapper[2579]: E0730 01:11:28.180229    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a75fd962cd958f5b3d702743a520a4a08035ebaa1396a5ec5e35b043d43998\": container with ID starting with 08a75fd962cd958f5b3d702743a520a4a08035ebaa1396a5ec5e35b043d43998 not found: ID does not exist" containerID="08a75fd962cd958f5b3d702743a520a4a08035ebaa1396a5ec5e35b043d43998"
Jul 30 01:11:28.180385 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.180350    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a75fd962cd958f5b3d702743a520a4a08035ebaa1396a5ec5e35b043d43998"} err="failed to get container status \"08a75fd962cd958f5b3d702743a520a4a08035ebaa1396a5ec5e35b043d43998\": rpc error: code = NotFound desc = could not find container \"08a75fd962cd958f5b3d702743a520a4a08035ebaa1396a5ec5e35b043d43998\": container with ID starting with 08a75fd962cd958f5b3d702743a520a4a08035ebaa1396a5ec5e35b043d43998 not found: ID does not exist"
Jul 30 01:11:28.180385 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.180368    2579 scope.go:117] "RemoveContainer" containerID="db394a19492656951fd9d588cc4644e54f6508152abd7000dd420a86f65265ea"
Jul 30 01:11:28.188805 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.188776    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4"]
Jul 30 01:11:28.189520 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.189502    2579 scope.go:117] "RemoveContainer" containerID="db394a19492656951fd9d588cc4644e54f6508152abd7000dd420a86f65265ea"
Jul 30 01:11:28.189755 ip-10-0-135-68 kubenswrapper[2579]: E0730 01:11:28.189735    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db394a19492656951fd9d588cc4644e54f6508152abd7000dd420a86f65265ea\": container with ID starting with db394a19492656951fd9d588cc4644e54f6508152abd7000dd420a86f65265ea not found: ID does not exist" containerID="db394a19492656951fd9d588cc4644e54f6508152abd7000dd420a86f65265ea"
Jul 30 01:11:28.189800 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.189763    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db394a19492656951fd9d588cc4644e54f6508152abd7000dd420a86f65265ea"} err="failed to get container status \"db394a19492656951fd9d588cc4644e54f6508152abd7000dd420a86f65265ea\": rpc error: code = NotFound desc = could not find container \"db394a19492656951fd9d588cc4644e54f6508152abd7000dd420a86f65265ea\": container with ID starting with db394a19492656951fd9d588cc4644e54f6508152abd7000dd420a86f65265ea not found: ID does not exist"
Jul 30 01:11:28.195824 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.195802    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ff67f5-86b6446748-pq2h4"]
Jul 30 01:11:28.208429 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.208407    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd"]
Jul 30 01:11:28.215606 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.215586    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ff67f5-d897b5ddd-xt7hd"]
Jul 30 01:11:28.228573 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.228552    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"]
Jul 30 01:11:28.232579 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:28.232558    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ff67f5-85d847457f-dj6vk"]
Jul 30 01:11:29.325599 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:29.325567    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17c51729-1ae6-446c-9008-1054653c7b2f" path="/var/lib/kubelet/pods/17c51729-1ae6-446c-9008-1054653c7b2f/volumes"
Jul 30 01:11:29.325975 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:29.325942    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60edf95f-501c-4cb3-af18-95d52c548bd9" path="/var/lib/kubelet/pods/60edf95f-501c-4cb3-af18-95d52c548bd9/volumes"
Jul 30 01:11:29.326286 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:29.326273    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bacf50c0-c60f-4404-a46e-109d921e1acb" path="/var/lib/kubelet/pods/bacf50c0-c60f-4404-a46e-109d921e1acb/volumes"
Jul 30 01:11:37.550044 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:37.550005    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5912cb-q5j2t"]
Jul 30 01:11:37.550535 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:37.550494    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60edf95f-501c-4cb3-af18-95d52c548bd9" containerName="maas-api"
Jul 30 01:11:37.550535 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:37.550508    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="60edf95f-501c-4cb3-af18-95d52c548bd9" containerName="maas-api"
Jul 30 01:11:37.550535 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:37.550520    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bacf50c0-c60f-4404-a46e-109d921e1acb" containerName="payload-pre-processing"
Jul 30 01:11:37.550535 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:37.550525    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bacf50c0-c60f-4404-a46e-109d921e1acb" containerName="payload-pre-processing"
Jul 30 01:11:37.550764 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:37.550548    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17c51729-1ae6-446c-9008-1054653c7b2f" containerName="payload-processing"
Jul 30 01:11:37.550764 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:37.550555    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="17c51729-1ae6-446c-9008-1054653c7b2f" containerName="payload-processing"
Jul 30 01:11:37.550764 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:37.550617    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bacf50c0-c60f-4404-a46e-109d921e1acb" containerName="payload-pre-processing"
Jul 30 01:11:37.550764 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:37.550625    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="17c51729-1ae6-446c-9008-1054653c7b2f" containerName="payload-processing"
Jul 30 01:11:37.550764 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:37.550634    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="60edf95f-501c-4cb3-af18-95d52c548bd9" containerName="maas-api"
Jul 30 01:11:37.553549 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:37.553532    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5912cb-q5j2t"
Jul 30 01:11:37.556465 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:37.556448    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v8q7j\""
Jul 30 01:11:37.561797 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:37.561769    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5912cb-q5j2t"]
Jul 30 01:11:37.675396 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:37.675366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca17d437-d173-40bc-b598-db732d40cf3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5912cb-q5j2t\" (UID: \"ca17d437-d173-40bc-b598-db732d40cf3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5912cb-q5j2t"
Jul 30 01:11:37.776080 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:37.776048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca17d437-d173-40bc-b598-db732d40cf3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5912cb-q5j2t\" (UID: \"ca17d437-d173-40bc-b598-db732d40cf3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5912cb-q5j2t"
Jul 30 01:11:37.776724 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:37.776698    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca17d437-d173-40bc-b598-db732d40cf3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5912cb-q5j2t\" (UID: \"ca17d437-d173-40bc-b598-db732d40cf3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5912cb-q5j2t"
Jul 30 01:11:37.865763 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:37.865688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5912cb-q5j2t"
Jul 30 01:11:38.210625 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:38.210600    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5912cb-q5j2t"]
Jul 30 01:11:38.212560 ip-10-0-135-68 kubenswrapper[2579]: W0730 01:11:38.212535    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca17d437_d173_40bc_b598_db732d40cf3f.slice/crio-4a2936fc2a0de22ffe167c0e02d7d537da54b4ec487095cd08da644931739818 WatchSource:0}: Error finding container 4a2936fc2a0de22ffe167c0e02d7d537da54b4ec487095cd08da644931739818: Status 404 returned error can't find the container with id 4a2936fc2a0de22ffe167c0e02d7d537da54b4ec487095cd08da644931739818
Jul 30 01:11:38.221801 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:38.221778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5912cb-q5j2t" event={"ID":"ca17d437-d173-40bc-b598-db732d40cf3f","Type":"ContainerStarted","Data":"4a2936fc2a0de22ffe167c0e02d7d537da54b4ec487095cd08da644931739818"}
Jul 30 01:11:39.228356 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:39.228234    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca17d437-d173-40bc-b598-db732d40cf3f" containerID="74ce7c1c11a6bef4168abaed22c481246846957e2de57c84067895ed16f152ae" exitCode=0
Jul 30 01:11:39.228356 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:39.228320    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5912cb-q5j2t" event={"ID":"ca17d437-d173-40bc-b598-db732d40cf3f","Type":"ContainerDied","Data":"74ce7c1c11a6bef4168abaed22c481246846957e2de57c84067895ed16f152ae"}
Jul 30 01:11:40.373027 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:40.372999    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5912cb-q5j2t"
Jul 30 01:11:40.505454 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:40.505365    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca17d437-d173-40bc-b598-db732d40cf3f-maas-api-service-ca\") pod \"ca17d437-d173-40bc-b598-db732d40cf3f\" (UID: \"ca17d437-d173-40bc-b598-db732d40cf3f\") "
Jul 30 01:11:40.505724 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:40.505702    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca17d437-d173-40bc-b598-db732d40cf3f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ca17d437-d173-40bc-b598-db732d40cf3f" (UID: "ca17d437-d173-40bc-b598-db732d40cf3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 01:11:40.606634 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:40.606601    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca17d437-d173-40bc-b598-db732d40cf3f-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:11:41.240312 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:41.240280    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5912cb-q5j2t"
Jul 30 01:11:41.240515 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:41.240280    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5912cb-q5j2t" event={"ID":"ca17d437-d173-40bc-b598-db732d40cf3f","Type":"ContainerDied","Data":"4a2936fc2a0de22ffe167c0e02d7d537da54b4ec487095cd08da644931739818"}
Jul 30 01:11:41.240515 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:41.240425    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a2936fc2a0de22ffe167c0e02d7d537da54b4ec487095cd08da644931739818"
Jul 30 01:11:50.986986 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:50.986902    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"]
Jul 30 01:11:50.987439 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:50.987153    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd" podUID="e8bc9a6d-e2e8-4a40-b02e-d27b75af3510" containerName="maas-api" containerID="cri-o://e2e0e9efe40e90f3c1f1cb777e15e9e1cf9aab2863239506f80c2e83eae4d26c" gracePeriod=30
Jul 30 01:11:51.046802 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.046774    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97"]
Jul 30 01:11:51.047090 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.047041    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97" podUID="84d7d024-fac5-4f01-ac91-36faf1a41cda" containerName="payload-pre-processing" containerID="cri-o://d61082d1c93e2f10102a404e67c62b1a403bdf167f36a62631f40e380dd3a14f" gracePeriod=30
Jul 30 01:11:51.104009 ip-10-0-135-68 kubenswrapper[2579]: E0730 01:11:51.103977    2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d7d024_fac5_4f01_ac91_36faf1a41cda.slice/crio-conmon-d61082d1c93e2f10102a404e67c62b1a403bdf167f36a62631f40e380dd3a14f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d7d024_fac5_4f01_ac91_36faf1a41cda.slice/crio-d61082d1c93e2f10102a404e67c62b1a403bdf167f36a62631f40e380dd3a14f.scope\": RecentStats: unable to find data in memory cache]"
Jul 30 01:11:51.104154 ip-10-0-135-68 kubenswrapper[2579]: E0730 01:11:51.104018    2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d7d024_fac5_4f01_ac91_36faf1a41cda.slice/crio-conmon-d61082d1c93e2f10102a404e67c62b1a403bdf167f36a62631f40e380dd3a14f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d7d024_fac5_4f01_ac91_36faf1a41cda.slice/crio-d61082d1c93e2f10102a404e67c62b1a403bdf167f36a62631f40e380dd3a14f.scope\": RecentStats: unable to find data in memory cache]"
Jul 30 01:11:51.174108 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.174075    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5912cb-q5j2t"]
Jul 30 01:11:51.176457 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.176432    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5912cb-q5j2t"]
Jul 30 01:11:51.302578 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.302550    2579 generic.go:356] "Generic (PLEG): container finished" podID="84d7d024-fac5-4f01-ac91-36faf1a41cda" containerID="d61082d1c93e2f10102a404e67c62b1a403bdf167f36a62631f40e380dd3a14f" exitCode=0
Jul 30 01:11:51.302717 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.302624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97" event={"ID":"84d7d024-fac5-4f01-ac91-36faf1a41cda","Type":"ContainerDied","Data":"d61082d1c93e2f10102a404e67c62b1a403bdf167f36a62631f40e380dd3a14f"}
Jul 30 01:11:51.304020 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.303995    2579 generic.go:356] "Generic (PLEG): container finished" podID="e8bc9a6d-e2e8-4a40-b02e-d27b75af3510" containerID="e2e0e9efe40e90f3c1f1cb777e15e9e1cf9aab2863239506f80c2e83eae4d26c" exitCode=0
Jul 30 01:11:51.304126 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.304049    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd" event={"ID":"e8bc9a6d-e2e8-4a40-b02e-d27b75af3510","Type":"ContainerDied","Data":"e2e0e9efe40e90f3c1f1cb777e15e9e1cf9aab2863239506f80c2e83eae4d26c"}
Jul 30 01:11:51.324601 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.324570    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca17d437-d173-40bc-b598-db732d40cf3f" path="/var/lib/kubelet/pods/ca17d437-d173-40bc-b598-db732d40cf3f/volumes"
Jul 30 01:11:51.332939 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.332921    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"
Jul 30 01:11:51.336218 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.336201    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97"
Jul 30 01:11:51.515354 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.515277    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8bc9a6d-e2e8-4a40-b02e-d27b75af3510-maas-api-tls\") pod \"e8bc9a6d-e2e8-4a40-b02e-d27b75af3510\" (UID: \"e8bc9a6d-e2e8-4a40-b02e-d27b75af3510\") "
Jul 30 01:11:51.515354 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.515326    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d7d024-fac5-4f01-ac91-36faf1a41cda-plugins-config-volume\") pod \"84d7d024-fac5-4f01-ac91-36faf1a41cda\" (UID: \"84d7d024-fac5-4f01-ac91-36faf1a41cda\") "
Jul 30 01:11:51.515354 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.515353    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkgtg\" (UniqueName: \"kubernetes.io/projected/e8bc9a6d-e2e8-4a40-b02e-d27b75af3510-kube-api-access-fkgtg\") pod \"e8bc9a6d-e2e8-4a40-b02e-d27b75af3510\" (UID: \"e8bc9a6d-e2e8-4a40-b02e-d27b75af3510\") "
Jul 30 01:11:51.515641 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.515397    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bc9a6d-e2e8-4a40-b02e-d27b75af3510-openshift-service-ca\") pod \"e8bc9a6d-e2e8-4a40-b02e-d27b75af3510\" (UID: \"e8bc9a6d-e2e8-4a40-b02e-d27b75af3510\") "
Jul 30 01:11:51.515641 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.515413    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stk8s\" (UniqueName: \"kubernetes.io/projected/84d7d024-fac5-4f01-ac91-36faf1a41cda-kube-api-access-stk8s\") pod \"84d7d024-fac5-4f01-ac91-36faf1a41cda\" (UID: \"84d7d024-fac5-4f01-ac91-36faf1a41cda\") "
Jul 30 01:11:51.515752 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.515719    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d7d024-fac5-4f01-ac91-36faf1a41cda-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84d7d024-fac5-4f01-ac91-36faf1a41cda" (UID: "84d7d024-fac5-4f01-ac91-36faf1a41cda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 01:11:51.515814 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.515779    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8bc9a6d-e2e8-4a40-b02e-d27b75af3510-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e8bc9a6d-e2e8-4a40-b02e-d27b75af3510" (UID: "e8bc9a6d-e2e8-4a40-b02e-d27b75af3510"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 01:11:51.515975 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.515895    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bc9a6d-e2e8-4a40-b02e-d27b75af3510-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:11:51.515975 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.515916    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d7d024-fac5-4f01-ac91-36faf1a41cda-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:11:51.517612 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.517593    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8bc9a6d-e2e8-4a40-b02e-d27b75af3510-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e8bc9a6d-e2e8-4a40-b02e-d27b75af3510" (UID: "e8bc9a6d-e2e8-4a40-b02e-d27b75af3510"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 01:11:51.518013 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.517987    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bc9a6d-e2e8-4a40-b02e-d27b75af3510-kube-api-access-fkgtg" (OuterVolumeSpecName: "kube-api-access-fkgtg") pod "e8bc9a6d-e2e8-4a40-b02e-d27b75af3510" (UID: "e8bc9a6d-e2e8-4a40-b02e-d27b75af3510"). InnerVolumeSpecName "kube-api-access-fkgtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 01:11:51.518013 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.518013    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d7d024-fac5-4f01-ac91-36faf1a41cda-kube-api-access-stk8s" (OuterVolumeSpecName: "kube-api-access-stk8s") pod "84d7d024-fac5-4f01-ac91-36faf1a41cda" (UID: "84d7d024-fac5-4f01-ac91-36faf1a41cda"). InnerVolumeSpecName "kube-api-access-stk8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 01:11:51.616749 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.616722    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8bc9a6d-e2e8-4a40-b02e-d27b75af3510-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:11:51.616749 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.616745    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkgtg\" (UniqueName: \"kubernetes.io/projected/e8bc9a6d-e2e8-4a40-b02e-d27b75af3510-kube-api-access-fkgtg\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:11:51.616918 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:51.616754    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stk8s\" (UniqueName: \"kubernetes.io/projected/84d7d024-fac5-4f01-ac91-36faf1a41cda-kube-api-access-stk8s\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:11:52.311208 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:52.311131    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97"
Jul 30 01:11:52.311208 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:52.311155    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97" event={"ID":"84d7d024-fac5-4f01-ac91-36faf1a41cda","Type":"ContainerDied","Data":"9c5fd94e517efdc5e83a878ce261c9df90c72f352fa2038ae41c663a424d43be"}
Jul 30 01:11:52.311721 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:52.311261    2579 scope.go:117] "RemoveContainer" containerID="d61082d1c93e2f10102a404e67c62b1a403bdf167f36a62631f40e380dd3a14f"
Jul 30 01:11:52.312641 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:52.312621    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd" event={"ID":"e8bc9a6d-e2e8-4a40-b02e-d27b75af3510","Type":"ContainerDied","Data":"4b47403d7822c71ca189948dd83cf70bf253281a7ac875279b3e08ce9734d8be"}
Jul 30 01:11:52.312710 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:52.312663    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"
Jul 30 01:11:52.321832 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:52.321814    2579 scope.go:117] "RemoveContainer" containerID="e2e0e9efe40e90f3c1f1cb777e15e9e1cf9aab2863239506f80c2e83eae4d26c"
Jul 30 01:11:52.335450 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:52.335426    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97"]
Jul 30 01:11:52.337444 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:52.337424    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5912cb-86f69988bd-s7c97"]
Jul 30 01:11:52.348602 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:52.348577    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"]
Jul 30 01:11:52.351577 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:52.351554    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5912cb-d944bbbcb-6zckd"]
Jul 30 01:11:53.322403 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:53.322365    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84d7d024-fac5-4f01-ac91-36faf1a41cda" path="/var/lib/kubelet/pods/84d7d024-fac5-4f01-ac91-36faf1a41cda/volumes"
Jul 30 01:11:53.322945 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:11:53.322924    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8bc9a6d-e2e8-4a40-b02e-d27b75af3510" path="/var/lib/kubelet/pods/e8bc9a6d-e2e8-4a40-b02e-d27b75af3510/volumes"
Jul 30 01:12:46.974616 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:46.974580    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47d629-2x7h9"]
Jul 30 01:12:46.975138 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:46.975034    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84d7d024-fac5-4f01-ac91-36faf1a41cda" containerName="payload-pre-processing"
Jul 30 01:12:46.975138 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:46.975045    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="84d7d024-fac5-4f01-ac91-36faf1a41cda" containerName="payload-pre-processing"
Jul 30 01:12:46.975138 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:46.975054    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8bc9a6d-e2e8-4a40-b02e-d27b75af3510" containerName="maas-api"
Jul 30 01:12:46.975138 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:46.975059    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8bc9a6d-e2e8-4a40-b02e-d27b75af3510" containerName="maas-api"
Jul 30 01:12:46.975138 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:46.975073    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca17d437-d173-40bc-b598-db732d40cf3f" containerName="revoke-keys"
Jul 30 01:12:46.975138 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:46.975079    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca17d437-d173-40bc-b598-db732d40cf3f" containerName="revoke-keys"
Jul 30 01:12:46.975138 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:46.975139    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8bc9a6d-e2e8-4a40-b02e-d27b75af3510" containerName="maas-api"
Jul 30 01:12:46.975396 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:46.975148    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="84d7d024-fac5-4f01-ac91-36faf1a41cda" containerName="payload-pre-processing"
Jul 30 01:12:46.975396 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:46.975155    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca17d437-d173-40bc-b598-db732d40cf3f" containerName="revoke-keys"
Jul 30 01:12:46.978078 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:46.978062    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47d629-2x7h9"
Jul 30 01:12:46.980747 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:46.980725    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v8q7j\""
Jul 30 01:12:46.986130 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:46.986019    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47d629-2x7h9"]
Jul 30 01:12:47.020633 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:47.020602    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa71380-9fb5-47a6-800b-4e6d1b81b862-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-47d629-2x7h9\" (UID: \"1aa71380-9fb5-47a6-800b-4e6d1b81b862\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47d629-2x7h9"
Jul 30 01:12:47.121956 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:47.121923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa71380-9fb5-47a6-800b-4e6d1b81b862-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-47d629-2x7h9\" (UID: \"1aa71380-9fb5-47a6-800b-4e6d1b81b862\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47d629-2x7h9"
Jul 30 01:12:47.122602 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:47.122576    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa71380-9fb5-47a6-800b-4e6d1b81b862-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-47d629-2x7h9\" (UID: \"1aa71380-9fb5-47a6-800b-4e6d1b81b862\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47d629-2x7h9"
Jul 30 01:12:47.290137 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:47.290064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47d629-2x7h9"
Jul 30 01:12:47.437626 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:47.437592    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47d629-2x7h9"]
Jul 30 01:12:47.438828 ip-10-0-135-68 kubenswrapper[2579]: W0730 01:12:47.438799    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa71380_9fb5_47a6_800b_4e6d1b81b862.slice/crio-8280f43cbdbfde22de48fd512f5c3972c4e92469cb710f36540b4525180eb463 WatchSource:0}: Error finding container 8280f43cbdbfde22de48fd512f5c3972c4e92469cb710f36540b4525180eb463: Status 404 returned error can't find the container with id 8280f43cbdbfde22de48fd512f5c3972c4e92469cb710f36540b4525180eb463
Jul 30 01:12:47.596414 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:47.596327    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47d629-2x7h9" event={"ID":"1aa71380-9fb5-47a6-800b-4e6d1b81b862","Type":"ContainerStarted","Data":"8280f43cbdbfde22de48fd512f5c3972c4e92469cb710f36540b4525180eb463"}
Jul 30 01:12:48.603328 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:48.603188    2579 generic.go:356] "Generic (PLEG): container finished" podID="1aa71380-9fb5-47a6-800b-4e6d1b81b862" containerID="693231ade9545dce4a66b26cc015580de5cbf212bf73aba4a230a77e4a6273eb" exitCode=0
Jul 30 01:12:48.603328 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:48.603275    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47d629-2x7h9" event={"ID":"1aa71380-9fb5-47a6-800b-4e6d1b81b862","Type":"ContainerDied","Data":"693231ade9545dce4a66b26cc015580de5cbf212bf73aba4a230a77e4a6273eb"}
Jul 30 01:12:49.743873 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:49.743847    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47d629-2x7h9"
Jul 30 01:12:49.847988 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:49.847953    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa71380-9fb5-47a6-800b-4e6d1b81b862-maas-api-service-ca\") pod \"1aa71380-9fb5-47a6-800b-4e6d1b81b862\" (UID: \"1aa71380-9fb5-47a6-800b-4e6d1b81b862\") "
Jul 30 01:12:49.848400 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:49.848372    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa71380-9fb5-47a6-800b-4e6d1b81b862-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1aa71380-9fb5-47a6-800b-4e6d1b81b862" (UID: "1aa71380-9fb5-47a6-800b-4e6d1b81b862"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 01:12:49.949548 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:49.949525    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa71380-9fb5-47a6-800b-4e6d1b81b862-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:12:50.615183 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:50.615140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47d629-2x7h9" event={"ID":"1aa71380-9fb5-47a6-800b-4e6d1b81b862","Type":"ContainerDied","Data":"8280f43cbdbfde22de48fd512f5c3972c4e92469cb710f36540b4525180eb463"}
Jul 30 01:12:50.615183 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:50.615179    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47d629-2x7h9"
Jul 30 01:12:50.615427 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:50.615204    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8280f43cbdbfde22de48fd512f5c3972c4e92469cb710f36540b4525180eb463"
Jul 30 01:12:53.129735 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.129687    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"]
Jul 30 01:12:53.130554 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.130518    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5" podUID="3d582f6c-cdf2-4880-bae4-f3c523131855" containerName="maas-api" containerID="cri-o://abca6f1c0f814a95fedc45af1ae2a777176296bd0dbe0fcd145caee2e3f03e7d" gracePeriod=30
Jul 30 01:12:53.388130 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.388069    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"
Jul 30 01:12:53.467978 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.467931    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47d629-2x7h9"]
Jul 30 01:12:53.469766 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.469745    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47d629-2x7h9"]
Jul 30 01:12:53.483993 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.483963    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d582f6c-cdf2-4880-bae4-f3c523131855-maas-api-tls\") pod \"3d582f6c-cdf2-4880-bae4-f3c523131855\" (UID: \"3d582f6c-cdf2-4880-bae4-f3c523131855\") "
Jul 30 01:12:53.484136 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.484034    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d582f6c-cdf2-4880-bae4-f3c523131855-openshift-service-ca\") pod \"3d582f6c-cdf2-4880-bae4-f3c523131855\" (UID: \"3d582f6c-cdf2-4880-bae4-f3c523131855\") "
Jul 30 01:12:53.484205 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.484176    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjr7m\" (UniqueName: \"kubernetes.io/projected/3d582f6c-cdf2-4880-bae4-f3c523131855-kube-api-access-qjr7m\") pod \"3d582f6c-cdf2-4880-bae4-f3c523131855\" (UID: \"3d582f6c-cdf2-4880-bae4-f3c523131855\") "
Jul 30 01:12:53.484594 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.484451    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d582f6c-cdf2-4880-bae4-f3c523131855-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3d582f6c-cdf2-4880-bae4-f3c523131855" (UID: "3d582f6c-cdf2-4880-bae4-f3c523131855"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 01:12:53.484594 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.484567    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d582f6c-cdf2-4880-bae4-f3c523131855-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:12:53.486440 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.486408    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d582f6c-cdf2-4880-bae4-f3c523131855-kube-api-access-qjr7m" (OuterVolumeSpecName: "kube-api-access-qjr7m") pod "3d582f6c-cdf2-4880-bae4-f3c523131855" (UID: "3d582f6c-cdf2-4880-bae4-f3c523131855"). InnerVolumeSpecName "kube-api-access-qjr7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 01:12:53.486642 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.486623    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d582f6c-cdf2-4880-bae4-f3c523131855-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3d582f6c-cdf2-4880-bae4-f3c523131855" (UID: "3d582f6c-cdf2-4880-bae4-f3c523131855"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 01:12:53.585418 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.585387    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjr7m\" (UniqueName: \"kubernetes.io/projected/3d582f6c-cdf2-4880-bae4-f3c523131855-kube-api-access-qjr7m\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:12:53.585418 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.585413    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d582f6c-cdf2-4880-bae4-f3c523131855-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:12:53.631809 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.631784    2579 generic.go:356] "Generic (PLEG): container finished" podID="3d582f6c-cdf2-4880-bae4-f3c523131855" containerID="abca6f1c0f814a95fedc45af1ae2a777176296bd0dbe0fcd145caee2e3f03e7d" exitCode=0
Jul 30 01:12:53.631922 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.631845    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"
Jul 30 01:12:53.631922 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.631877    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5" event={"ID":"3d582f6c-cdf2-4880-bae4-f3c523131855","Type":"ContainerDied","Data":"abca6f1c0f814a95fedc45af1ae2a777176296bd0dbe0fcd145caee2e3f03e7d"}
Jul 30 01:12:53.631993 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.631931    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5" event={"ID":"3d582f6c-cdf2-4880-bae4-f3c523131855","Type":"ContainerDied","Data":"33df11b19248e9ad8be10e7582ee9bacb1fcc538b6c7cee26358686a70cddf20"}
Jul 30 01:12:53.631993 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.631957    2579 scope.go:117] "RemoveContainer" containerID="abca6f1c0f814a95fedc45af1ae2a777176296bd0dbe0fcd145caee2e3f03e7d"
Jul 30 01:12:53.641084 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.641066    2579 scope.go:117] "RemoveContainer" containerID="abca6f1c0f814a95fedc45af1ae2a777176296bd0dbe0fcd145caee2e3f03e7d"
Jul 30 01:12:53.641369 ip-10-0-135-68 kubenswrapper[2579]: E0730 01:12:53.641350    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abca6f1c0f814a95fedc45af1ae2a777176296bd0dbe0fcd145caee2e3f03e7d\": container with ID starting with abca6f1c0f814a95fedc45af1ae2a777176296bd0dbe0fcd145caee2e3f03e7d not found: ID does not exist" containerID="abca6f1c0f814a95fedc45af1ae2a777176296bd0dbe0fcd145caee2e3f03e7d"
Jul 30 01:12:53.641429 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.641378    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abca6f1c0f814a95fedc45af1ae2a777176296bd0dbe0fcd145caee2e3f03e7d"} err="failed to get container status \"abca6f1c0f814a95fedc45af1ae2a777176296bd0dbe0fcd145caee2e3f03e7d\": rpc error: code = NotFound desc = could not find container \"abca6f1c0f814a95fedc45af1ae2a777176296bd0dbe0fcd145caee2e3f03e7d\": container with ID starting with abca6f1c0f814a95fedc45af1ae2a777176296bd0dbe0fcd145caee2e3f03e7d not found: ID does not exist"
Jul 30 01:12:53.653856 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.653835    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"]
Jul 30 01:12:53.657163 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:53.657143    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-47d629-66c86894b5-hr5m5"]
Jul 30 01:12:55.323533 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:55.323500    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aa71380-9fb5-47a6-800b-4e6d1b81b862" path="/var/lib/kubelet/pods/1aa71380-9fb5-47a6-800b-4e6d1b81b862/volumes"
Jul 30 01:12:55.323901 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:55.323880    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d582f6c-cdf2-4880-bae4-f3c523131855" path="/var/lib/kubelet/pods/3d582f6c-cdf2-4880-bae4-f3c523131855/volumes"
Jul 30 01:12:55.463271 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:55.463231    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 01:12:55.470996 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:12:55.470977    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-slhpw_fc610fab-9658-40eb-8527-081a0008d70f/ovn-acl-logging/0.log"
Jul 30 01:13:04.831797 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:04.831761    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"]
Jul 30 01:13:04.832413 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:04.832022    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q" podUID="eaeec7a0-65a9-4179-b8e2-9fc07d847f5a" containerName="maas-api" containerID="cri-o://509e306a19074348ce7a1051f4ca3e81395ef276da9338bc934258ab6f04d4c5" gracePeriod=30
Jul 30 01:13:04.890264 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:04.890221    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8"]
Jul 30 01:13:04.890491 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:04.890447    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8" podUID="1ee3a598-1308-44de-9775-86a0b4bda898" containerName="payload-processing" containerID="cri-o://dded4ac1c7c056906edafa25e4a900ddbbb923ebb39a145acde5afec6e9f8f6f" gracePeriod=30
Jul 30 01:13:04.896325 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:04.896294    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm"]
Jul 30 01:13:04.896980 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:04.896517    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm" podUID="f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e" containerName="payload-pre-processing" containerID="cri-o://7b232827babcbfcb57bdadbf482b220fcfb4116ee59c9377c3370163b4b464b7" gracePeriod=30
Jul 30 01:13:05.169427 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.169393    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8"
Jul 30 01:13:05.191619 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.191589    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm"
Jul 30 01:13:05.201193 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.201169    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"
Jul 30 01:13:05.288062 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.288035    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaeec7a0-65a9-4179-b8e2-9fc07d847f5a-openshift-service-ca\") pod \"eaeec7a0-65a9-4179-b8e2-9fc07d847f5a\" (UID: \"eaeec7a0-65a9-4179-b8e2-9fc07d847f5a\") "
Jul 30 01:13:05.288275 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.288069    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbr42\" (UniqueName: \"kubernetes.io/projected/eaeec7a0-65a9-4179-b8e2-9fc07d847f5a-kube-api-access-rbr42\") pod \"eaeec7a0-65a9-4179-b8e2-9fc07d847f5a\" (UID: \"eaeec7a0-65a9-4179-b8e2-9fc07d847f5a\") "
Jul 30 01:13:05.288275 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.288096    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsdwq\" (UniqueName: \"kubernetes.io/projected/f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e-kube-api-access-dsdwq\") pod \"f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e\" (UID: \"f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e\") "
Jul 30 01:13:05.288275 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.288123    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e-plugins-config-volume\") pod \"f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e\" (UID: \"f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e\") "
Jul 30 01:13:05.288465 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.288281    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaeec7a0-65a9-4179-b8e2-9fc07d847f5a-maas-api-tls\") pod \"eaeec7a0-65a9-4179-b8e2-9fc07d847f5a\" (UID: \"eaeec7a0-65a9-4179-b8e2-9fc07d847f5a\") "
Jul 30 01:13:05.288465 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.288366    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee3a598-1308-44de-9775-86a0b4bda898-plugins-config-volume\") pod \"1ee3a598-1308-44de-9775-86a0b4bda898\" (UID: \"1ee3a598-1308-44de-9775-86a0b4bda898\") "
Jul 30 01:13:05.288465 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.288409    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6x54\" (UniqueName: \"kubernetes.io/projected/1ee3a598-1308-44de-9775-86a0b4bda898-kube-api-access-n6x54\") pod \"1ee3a598-1308-44de-9775-86a0b4bda898\" (UID: \"1ee3a598-1308-44de-9775-86a0b4bda898\") "
Jul 30 01:13:05.288626 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.288527    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaeec7a0-65a9-4179-b8e2-9fc07d847f5a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eaeec7a0-65a9-4179-b8e2-9fc07d847f5a" (UID: "eaeec7a0-65a9-4179-b8e2-9fc07d847f5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 01:13:05.288626 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.288557    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e" (UID: "f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 01:13:05.288787 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.288758    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ee3a598-1308-44de-9775-86a0b4bda898-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ee3a598-1308-44de-9775-86a0b4bda898" (UID: "1ee3a598-1308-44de-9775-86a0b4bda898"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 01:13:05.288903 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.288877    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaeec7a0-65a9-4179-b8e2-9fc07d847f5a-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:13:05.288903 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.288901    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:13:05.289019 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.288915    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee3a598-1308-44de-9775-86a0b4bda898-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:13:05.290307 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.290283    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e-kube-api-access-dsdwq" (OuterVolumeSpecName: "kube-api-access-dsdwq") pod "f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e" (UID: "f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e"). InnerVolumeSpecName "kube-api-access-dsdwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 01:13:05.290538 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.290505    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaeec7a0-65a9-4179-b8e2-9fc07d847f5a-kube-api-access-rbr42" (OuterVolumeSpecName: "kube-api-access-rbr42") pod "eaeec7a0-65a9-4179-b8e2-9fc07d847f5a" (UID: "eaeec7a0-65a9-4179-b8e2-9fc07d847f5a"). InnerVolumeSpecName "kube-api-access-rbr42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 01:13:05.290718 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.290699    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaeec7a0-65a9-4179-b8e2-9fc07d847f5a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eaeec7a0-65a9-4179-b8e2-9fc07d847f5a" (UID: "eaeec7a0-65a9-4179-b8e2-9fc07d847f5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 01:13:05.290809 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.290791    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee3a598-1308-44de-9775-86a0b4bda898-kube-api-access-n6x54" (OuterVolumeSpecName: "kube-api-access-n6x54") pod "1ee3a598-1308-44de-9775-86a0b4bda898" (UID: "1ee3a598-1308-44de-9775-86a0b4bda898"). InnerVolumeSpecName "kube-api-access-n6x54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 01:13:05.389627 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.389602    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbr42\" (UniqueName: \"kubernetes.io/projected/eaeec7a0-65a9-4179-b8e2-9fc07d847f5a-kube-api-access-rbr42\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:13:05.389738 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.389629    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsdwq\" (UniqueName: \"kubernetes.io/projected/f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e-kube-api-access-dsdwq\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:13:05.389738 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.389639    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaeec7a0-65a9-4179-b8e2-9fc07d847f5a-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:13:05.389738 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.389648    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6x54\" (UniqueName: \"kubernetes.io/projected/1ee3a598-1308-44de-9775-86a0b4bda898-kube-api-access-n6x54\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:13:05.698414 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.698333    2579 generic.go:356] "Generic (PLEG): container finished" podID="f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e" containerID="7b232827babcbfcb57bdadbf482b220fcfb4116ee59c9377c3370163b4b464b7" exitCode=0
Jul 30 01:13:05.698414 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.698375    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm"
Jul 30 01:13:05.698613 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.698420    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm" event={"ID":"f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e","Type":"ContainerDied","Data":"7b232827babcbfcb57bdadbf482b220fcfb4116ee59c9377c3370163b4b464b7"}
Jul 30 01:13:05.698613 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.698472    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm" event={"ID":"f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e","Type":"ContainerDied","Data":"ebbd1c8f1b4bd9197faee17f8301b632e7173e24e380d8fed24e20d6752b537c"}
Jul 30 01:13:05.699565 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.698499    2579 scope.go:117] "RemoveContainer" containerID="7b232827babcbfcb57bdadbf482b220fcfb4116ee59c9377c3370163b4b464b7"
Jul 30 01:13:05.699858 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.699826    2579 generic.go:356] "Generic (PLEG): container finished" podID="eaeec7a0-65a9-4179-b8e2-9fc07d847f5a" containerID="509e306a19074348ce7a1051f4ca3e81395ef276da9338bc934258ab6f04d4c5" exitCode=0
Jul 30 01:13:05.699858 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.699858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q" event={"ID":"eaeec7a0-65a9-4179-b8e2-9fc07d847f5a","Type":"ContainerDied","Data":"509e306a19074348ce7a1051f4ca3e81395ef276da9338bc934258ab6f04d4c5"}
Jul 30 01:13:05.700031 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.699871    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"
Jul 30 01:13:05.700031 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.699921    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q" event={"ID":"eaeec7a0-65a9-4179-b8e2-9fc07d847f5a","Type":"ContainerDied","Data":"290532e33a954e10fbb1e82a4c13fa0b308cd0953bf94ee36a31034a12ff7a87"}
Jul 30 01:13:05.701751 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.701727    2579 generic.go:356] "Generic (PLEG): container finished" podID="1ee3a598-1308-44de-9775-86a0b4bda898" containerID="dded4ac1c7c056906edafa25e4a900ddbbb923ebb39a145acde5afec6e9f8f6f" exitCode=0
Jul 30 01:13:05.701872 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.701775    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8"
Jul 30 01:13:05.701872 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.701801    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8" event={"ID":"1ee3a598-1308-44de-9775-86a0b4bda898","Type":"ContainerDied","Data":"dded4ac1c7c056906edafa25e4a900ddbbb923ebb39a145acde5afec6e9f8f6f"}
Jul 30 01:13:05.701872 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.701838    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8" event={"ID":"1ee3a598-1308-44de-9775-86a0b4bda898","Type":"ContainerDied","Data":"5d749a99f95651117f6e0e56400afd5f5b90674b3529d6d36c09f0b1fb239c5c"}
Jul 30 01:13:05.709418 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.709306    2579 scope.go:117] "RemoveContainer" containerID="7b232827babcbfcb57bdadbf482b220fcfb4116ee59c9377c3370163b4b464b7"
Jul 30 01:13:05.709580 ip-10-0-135-68 kubenswrapper[2579]: E0730 01:13:05.709563    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b232827babcbfcb57bdadbf482b220fcfb4116ee59c9377c3370163b4b464b7\": container with ID starting with 7b232827babcbfcb57bdadbf482b220fcfb4116ee59c9377c3370163b4b464b7 not found: ID does not exist" containerID="7b232827babcbfcb57bdadbf482b220fcfb4116ee59c9377c3370163b4b464b7"
Jul 30 01:13:05.709623 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.709589    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b232827babcbfcb57bdadbf482b220fcfb4116ee59c9377c3370163b4b464b7"} err="failed to get container status \"7b232827babcbfcb57bdadbf482b220fcfb4116ee59c9377c3370163b4b464b7\": rpc error: code = NotFound desc = could not find container \"7b232827babcbfcb57bdadbf482b220fcfb4116ee59c9377c3370163b4b464b7\": container with ID starting with 7b232827babcbfcb57bdadbf482b220fcfb4116ee59c9377c3370163b4b464b7 not found: ID does not exist"
Jul 30 01:13:05.709623 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.709607    2579 scope.go:117] "RemoveContainer" containerID="509e306a19074348ce7a1051f4ca3e81395ef276da9338bc934258ab6f04d4c5"
Jul 30 01:13:05.717991 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.717971    2579 scope.go:117] "RemoveContainer" containerID="509e306a19074348ce7a1051f4ca3e81395ef276da9338bc934258ab6f04d4c5"
Jul 30 01:13:05.718231 ip-10-0-135-68 kubenswrapper[2579]: E0730 01:13:05.718213    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"509e306a19074348ce7a1051f4ca3e81395ef276da9338bc934258ab6f04d4c5\": container with ID starting with 509e306a19074348ce7a1051f4ca3e81395ef276da9338bc934258ab6f04d4c5 not found: ID does not exist" containerID="509e306a19074348ce7a1051f4ca3e81395ef276da9338bc934258ab6f04d4c5"
Jul 30 01:13:05.718291 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.718236    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509e306a19074348ce7a1051f4ca3e81395ef276da9338bc934258ab6f04d4c5"} err="failed to get container status \"509e306a19074348ce7a1051f4ca3e81395ef276da9338bc934258ab6f04d4c5\": rpc error: code = NotFound desc = could not find container \"509e306a19074348ce7a1051f4ca3e81395ef276da9338bc934258ab6f04d4c5\": container with ID starting with 509e306a19074348ce7a1051f4ca3e81395ef276da9338bc934258ab6f04d4c5 not found: ID does not exist"
Jul 30 01:13:05.718291 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.718272    2579 scope.go:117] "RemoveContainer" containerID="dded4ac1c7c056906edafa25e4a900ddbbb923ebb39a145acde5afec6e9f8f6f"
Jul 30 01:13:05.722653 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.722633    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"]
Jul 30 01:13:05.726069 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.726050    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a47fe8-6cdcff9c5f-5ps8q"]
Jul 30 01:13:05.726283 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.726232    2579 scope.go:117] "RemoveContainer" containerID="dded4ac1c7c056906edafa25e4a900ddbbb923ebb39a145acde5afec6e9f8f6f"
Jul 30 01:13:05.726539 ip-10-0-135-68 kubenswrapper[2579]: E0730 01:13:05.726514    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dded4ac1c7c056906edafa25e4a900ddbbb923ebb39a145acde5afec6e9f8f6f\": container with ID starting with dded4ac1c7c056906edafa25e4a900ddbbb923ebb39a145acde5afec6e9f8f6f not found: ID does not exist" containerID="dded4ac1c7c056906edafa25e4a900ddbbb923ebb39a145acde5afec6e9f8f6f"
Jul 30 01:13:05.726631 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.726544    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dded4ac1c7c056906edafa25e4a900ddbbb923ebb39a145acde5afec6e9f8f6f"} err="failed to get container status \"dded4ac1c7c056906edafa25e4a900ddbbb923ebb39a145acde5afec6e9f8f6f\": rpc error: code = NotFound desc = could not find container \"dded4ac1c7c056906edafa25e4a900ddbbb923ebb39a145acde5afec6e9f8f6f\": container with ID starting with dded4ac1c7c056906edafa25e4a900ddbbb923ebb39a145acde5afec6e9f8f6f not found: ID does not exist"
Jul 30 01:13:05.735694 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.735674    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm"]
Jul 30 01:13:05.739044 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.739023    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a47fe8-6b4764cd7b-vrsmm"]
Jul 30 01:13:05.749407 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.749380    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8"]
Jul 30 01:13:05.751584 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:05.751564    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a47fe8-86dc5cf8db-cdkm8"]
Jul 30 01:13:07.323949 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:07.323909    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ee3a598-1308-44de-9775-86a0b4bda898" path="/var/lib/kubelet/pods/1ee3a598-1308-44de-9775-86a0b4bda898/volumes"
Jul 30 01:13:07.324378 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:07.324362    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaeec7a0-65a9-4179-b8e2-9fc07d847f5a" path="/var/lib/kubelet/pods/eaeec7a0-65a9-4179-b8e2-9fc07d847f5a/volumes"
Jul 30 01:13:07.324690 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:07.324676    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e" path="/var/lib/kubelet/pods/f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e/volumes"
Jul 30 01:13:17.491233 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:17.489873    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"]
Jul 30 01:13:17.491233 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:17.490174    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz" podUID="8e6fe840-95ea-4f5d-be96-55751b512705" containerName="maas-api" containerID="cri-o://453bcf8e68210224ea6bbff2d4a69f2a8e88f1f877856ee291738f4bb2a635b5" gracePeriod=30
Jul 30 01:13:17.562616 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:17.562580    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx"]
Jul 30 01:13:17.563198 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:17.562826    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx" podUID="637a7f86-1d4e-419a-aea6-bdaff9d4044c" containerName="payload-processing" containerID="cri-o://865abd9a9dd5bbaed47b61908190ae312ec7daedbb924efb7948efd26f06b07e" gracePeriod=30
Jul 30 01:13:17.578382 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:17.578354    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl"]
Jul 30 01:13:17.578633 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:17.578607    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl" podUID="3f3ee87d-985d-4983-8609-c7feb4e2970d" containerName="payload-pre-processing" containerID="cri-o://568f68876907d6b6dd0989b504df6d38a7204a519cad75f7a94c8a787e251faf" gracePeriod=30
Jul 30 01:13:17.772380 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:17.772348    2579 generic.go:356] "Generic (PLEG): container finished" podID="637a7f86-1d4e-419a-aea6-bdaff9d4044c" containerID="865abd9a9dd5bbaed47b61908190ae312ec7daedbb924efb7948efd26f06b07e" exitCode=0
Jul 30 01:13:17.772600 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:17.772425    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx" event={"ID":"637a7f86-1d4e-419a-aea6-bdaff9d4044c","Type":"ContainerDied","Data":"865abd9a9dd5bbaed47b61908190ae312ec7daedbb924efb7948efd26f06b07e"}
Jul 30 01:13:17.774129 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:17.774107    2579 generic.go:356] "Generic (PLEG): container finished" podID="3f3ee87d-985d-4983-8609-c7feb4e2970d" containerID="568f68876907d6b6dd0989b504df6d38a7204a519cad75f7a94c8a787e251faf" exitCode=0
Jul 30 01:13:17.774253 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:17.774167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl" event={"ID":"3f3ee87d-985d-4983-8609-c7feb4e2970d","Type":"ContainerDied","Data":"568f68876907d6b6dd0989b504df6d38a7204a519cad75f7a94c8a787e251faf"}
Jul 30 01:13:17.775801 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:17.775779    2579 generic.go:356] "Generic (PLEG): container finished" podID="8e6fe840-95ea-4f5d-be96-55751b512705" containerID="453bcf8e68210224ea6bbff2d4a69f2a8e88f1f877856ee291738f4bb2a635b5" exitCode=0
Jul 30 01:13:17.775891 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:17.775847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz" event={"ID":"8e6fe840-95ea-4f5d-be96-55751b512705","Type":"ContainerDied","Data":"453bcf8e68210224ea6bbff2d4a69f2a8e88f1f877856ee291738f4bb2a635b5"}
Jul 30 01:13:17.862577 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:17.862550    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx"
Jul 30 01:13:17.865837 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:17.865815    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"
Jul 30 01:13:17.872810 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:17.872790    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl"
Jul 30 01:13:18.001913 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.001831    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637a7f86-1d4e-419a-aea6-bdaff9d4044c-plugins-config-volume\") pod \"637a7f86-1d4e-419a-aea6-bdaff9d4044c\" (UID: \"637a7f86-1d4e-419a-aea6-bdaff9d4044c\") "
Jul 30 01:13:18.001913 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.001877    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e6fe840-95ea-4f5d-be96-55751b512705-maas-api-tls\") pod \"8e6fe840-95ea-4f5d-be96-55751b512705\" (UID: \"8e6fe840-95ea-4f5d-be96-55751b512705\") "
Jul 30 01:13:18.001913 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.001906    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbfm8\" (UniqueName: \"kubernetes.io/projected/3f3ee87d-985d-4983-8609-c7feb4e2970d-kube-api-access-xbfm8\") pod \"3f3ee87d-985d-4983-8609-c7feb4e2970d\" (UID: \"3f3ee87d-985d-4983-8609-c7feb4e2970d\") "
Jul 30 01:13:18.002208 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.001929    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6fe840-95ea-4f5d-be96-55751b512705-openshift-service-ca\") pod \"8e6fe840-95ea-4f5d-be96-55751b512705\" (UID: \"8e6fe840-95ea-4f5d-be96-55751b512705\") "
Jul 30 01:13:18.002208 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.001979    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3ee87d-985d-4983-8609-c7feb4e2970d-plugins-config-volume\") pod \"3f3ee87d-985d-4983-8609-c7feb4e2970d\" (UID: \"3f3ee87d-985d-4983-8609-c7feb4e2970d\") "
Jul 30 01:13:18.002208 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.002051    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g99x4\" (UniqueName: \"kubernetes.io/projected/8e6fe840-95ea-4f5d-be96-55751b512705-kube-api-access-g99x4\") pod \"8e6fe840-95ea-4f5d-be96-55751b512705\" (UID: \"8e6fe840-95ea-4f5d-be96-55751b512705\") "
Jul 30 01:13:18.002208 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.002084    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jvkq\" (UniqueName: \"kubernetes.io/projected/637a7f86-1d4e-419a-aea6-bdaff9d4044c-kube-api-access-9jvkq\") pod \"637a7f86-1d4e-419a-aea6-bdaff9d4044c\" (UID: \"637a7f86-1d4e-419a-aea6-bdaff9d4044c\") "
Jul 30 01:13:18.002476 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.002438    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6fe840-95ea-4f5d-be96-55751b512705-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8e6fe840-95ea-4f5d-be96-55751b512705" (UID: "8e6fe840-95ea-4f5d-be96-55751b512705"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 01:13:18.002539 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.002511    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3ee87d-985d-4983-8609-c7feb4e2970d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f3ee87d-985d-4983-8609-c7feb4e2970d" (UID: "3f3ee87d-985d-4983-8609-c7feb4e2970d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 01:13:18.002598 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.002549    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6fe840-95ea-4f5d-be96-55751b512705-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:13:18.002665 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.002629    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/637a7f86-1d4e-419a-aea6-bdaff9d4044c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "637a7f86-1d4e-419a-aea6-bdaff9d4044c" (UID: "637a7f86-1d4e-419a-aea6-bdaff9d4044c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 01:13:18.004208 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.004171    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3ee87d-985d-4983-8609-c7feb4e2970d-kube-api-access-xbfm8" (OuterVolumeSpecName: "kube-api-access-xbfm8") pod "3f3ee87d-985d-4983-8609-c7feb4e2970d" (UID: "3f3ee87d-985d-4983-8609-c7feb4e2970d"). InnerVolumeSpecName "kube-api-access-xbfm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 01:13:18.004381 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.004345    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6fe840-95ea-4f5d-be96-55751b512705-kube-api-access-g99x4" (OuterVolumeSpecName: "kube-api-access-g99x4") pod "8e6fe840-95ea-4f5d-be96-55751b512705" (UID: "8e6fe840-95ea-4f5d-be96-55751b512705"). InnerVolumeSpecName "kube-api-access-g99x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 01:13:18.004625 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.004609    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/637a7f86-1d4e-419a-aea6-bdaff9d4044c-kube-api-access-9jvkq" (OuterVolumeSpecName: "kube-api-access-9jvkq") pod "637a7f86-1d4e-419a-aea6-bdaff9d4044c" (UID: "637a7f86-1d4e-419a-aea6-bdaff9d4044c"). InnerVolumeSpecName "kube-api-access-9jvkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 01:13:18.004690 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.004626    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6fe840-95ea-4f5d-be96-55751b512705-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8e6fe840-95ea-4f5d-be96-55751b512705" (UID: "8e6fe840-95ea-4f5d-be96-55751b512705"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 01:13:18.103617 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.103589    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbfm8\" (UniqueName: \"kubernetes.io/projected/3f3ee87d-985d-4983-8609-c7feb4e2970d-kube-api-access-xbfm8\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:13:18.103617 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.103612    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3ee87d-985d-4983-8609-c7feb4e2970d-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:13:18.103617 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.103622    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g99x4\" (UniqueName: \"kubernetes.io/projected/8e6fe840-95ea-4f5d-be96-55751b512705-kube-api-access-g99x4\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:13:18.103830 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.103631    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jvkq\" (UniqueName: \"kubernetes.io/projected/637a7f86-1d4e-419a-aea6-bdaff9d4044c-kube-api-access-9jvkq\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:13:18.103830 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.103639    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637a7f86-1d4e-419a-aea6-bdaff9d4044c-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:13:18.103830 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.103649    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e6fe840-95ea-4f5d-be96-55751b512705-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\""
Jul 30 01:13:18.783315 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.783207    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl"
Jul 30 01:13:18.783315 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.783215    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl" event={"ID":"3f3ee87d-985d-4983-8609-c7feb4e2970d","Type":"ContainerDied","Data":"b5661218e8685e1fa1493fb51eb7cfd5f86deca021df47283b8e850c0e243ede"}
Jul 30 01:13:18.783816 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.783389    2579 scope.go:117] "RemoveContainer" containerID="568f68876907d6b6dd0989b504df6d38a7204a519cad75f7a94c8a787e251faf"
Jul 30 01:13:18.784698 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.784680    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"
Jul 30 01:13:18.784768 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.784697    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz" event={"ID":"8e6fe840-95ea-4f5d-be96-55751b512705","Type":"ContainerDied","Data":"2bae5a9572432a4bbec3b2825da83f073834b16a900b946763c85d4d42143562"}
Jul 30 01:13:18.786188 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.786164    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx" event={"ID":"637a7f86-1d4e-419a-aea6-bdaff9d4044c","Type":"ContainerDied","Data":"0e22580a0429d031b40796ae37bc1dd0c717e1748af347b210d8b72cee3006b1"}
Jul 30 01:13:18.786319 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.786209    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx"
Jul 30 01:13:18.794724 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.794704    2579 scope.go:117] "RemoveContainer" containerID="453bcf8e68210224ea6bbff2d4a69f2a8e88f1f877856ee291738f4bb2a635b5"
Jul 30 01:13:18.804966 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.804950    2579 scope.go:117] "RemoveContainer" containerID="865abd9a9dd5bbaed47b61908190ae312ec7daedbb924efb7948efd26f06b07e"
Jul 30 01:13:18.814146 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.814125    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"]
Jul 30 01:13:18.820461 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.820440    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c4c410-c4848cd64-fw4sz"]
Jul 30 01:13:18.831660 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.831637    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl"]
Jul 30 01:13:18.835524 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.835502    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c4c410-566f8f4d6d-b5xvl"]
Jul 30 01:13:18.847326 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.847291    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx"]
Jul 30 01:13:18.854349 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:18.854328    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-c4c410-7b7d4d74c8-wcfrx"]
Jul 30 01:13:19.323138 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:19.323109    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f3ee87d-985d-4983-8609-c7feb4e2970d" path="/var/lib/kubelet/pods/3f3ee87d-985d-4983-8609-c7feb4e2970d/volumes"
Jul 30 01:13:19.323482 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:19.323468    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="637a7f86-1d4e-419a-aea6-bdaff9d4044c" path="/var/lib/kubelet/pods/637a7f86-1d4e-419a-aea6-bdaff9d4044c/volumes"
Jul 30 01:13:19.323821 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:19.323799    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e6fe840-95ea-4f5d-be96-55751b512705" path="/var/lib/kubelet/pods/8e6fe840-95ea-4f5d-be96-55751b512705/volumes"
Jul 30 01:13:30.507639 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:30.507605    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bd495bbc5-jvmkl_6440c0bc-07f7-4dfd-af29-c014028ecd2a/manager/0.log"
Jul 30 01:13:31.112823 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:31.112796    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b74bc5486-z878b_27f4f29e-3606-46f6-aaa6-6bae1807ace8/copy-manifests/0.log"
Jul 30 01:13:31.125964 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:31.125937    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b74bc5486-z878b_27f4f29e-3606-46f6-aaa6-6bae1807ace8/manager/0.log"
Jul 30 01:13:31.452166 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:31.452144    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-68d9bb9c54-bh2mr_57b0496d-ddb4-4f0f-8684-f8031487b466/maas-api/0.log"
Jul 30 01:13:31.664363 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:31.664334    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756205-pr8xl_fa35fdf6-d72f-4132-8a4c-5486c5f819d2/cleanup/0.log"
Jul 30 01:13:31.872650 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:31.872625    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-zpd9x_83d6e9fd-c623-479a-94fc-07134cdd40b8/postgres/0.log"
Jul 30 01:13:33.403828 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:33.403804    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-h8mmx_7fbd644a-580f-44e0-89cf-121ac1eb0c80/kuadrant-console-plugin/0.log"
Jul 30 01:13:34.487036 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:34.487009    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-94d649dc8-lm6p2_fb873962-c767-48e6-b6b3-09435499efa5/kube-auth-proxy/0.log"
Jul 30 01:13:34.703094 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:34.703062    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-m7n6j_9fc56b79-4db8-4489-a8c5-95e6349ce60d/payload-pre-processing/0.log"
Jul 30 01:13:34.807643 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:34.807570    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9k8fv_2e048fda-d493-49d2-955e-a4f1bb2a7f8e/payload-processing/0.log"
Jul 30 01:13:35.459194 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:35.459160    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-nftqr_ba605987-e2b2-4e1c-98c2-f8cfb8b6e17b/main/0.log"
Jul 30 01:13:35.785616 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:35.785540    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-g9rsw_afd1c313-29f1-4742-b3a7-d46a063bc081/main/0.log"
Jul 30 01:13:41.459651 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.459620    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p9qsq/must-gather-v9glv"]
Jul 30 01:13:41.460042 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.459985    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e6fe840-95ea-4f5d-be96-55751b512705" containerName="maas-api"
Jul 30 01:13:41.460042 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.459996    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e6fe840-95ea-4f5d-be96-55751b512705" containerName="maas-api"
Jul 30 01:13:41.460042 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460007    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aa71380-9fb5-47a6-800b-4e6d1b81b862" containerName="revoke-keys"
Jul 30 01:13:41.460042 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460013    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aa71380-9fb5-47a6-800b-4e6d1b81b862" containerName="revoke-keys"
Jul 30 01:13:41.460042 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460025    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="637a7f86-1d4e-419a-aea6-bdaff9d4044c" containerName="payload-processing"
Jul 30 01:13:41.460042 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460030    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="637a7f86-1d4e-419a-aea6-bdaff9d4044c" containerName="payload-processing"
Jul 30 01:13:41.460042 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460041    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e" containerName="payload-pre-processing"
Jul 30 01:13:41.460042 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460046    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e" containerName="payload-pre-processing"
Jul 30 01:13:41.460334 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460051    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3ee87d-985d-4983-8609-c7feb4e2970d" containerName="payload-pre-processing"
Jul 30 01:13:41.460334 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460058    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3ee87d-985d-4983-8609-c7feb4e2970d" containerName="payload-pre-processing"
Jul 30 01:13:41.460334 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460064    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d582f6c-cdf2-4880-bae4-f3c523131855" containerName="maas-api"
Jul 30 01:13:41.460334 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460069    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d582f6c-cdf2-4880-bae4-f3c523131855" containerName="maas-api"
Jul 30 01:13:41.460334 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460078    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ee3a598-1308-44de-9775-86a0b4bda898" containerName="payload-processing"
Jul 30 01:13:41.460334 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460083    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ee3a598-1308-44de-9775-86a0b4bda898" containerName="payload-processing"
Jul 30 01:13:41.460334 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460090    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaeec7a0-65a9-4179-b8e2-9fc07d847f5a" containerName="maas-api"
Jul 30 01:13:41.460334 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460095    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaeec7a0-65a9-4179-b8e2-9fc07d847f5a" containerName="maas-api"
Jul 30 01:13:41.460334 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460152    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaeec7a0-65a9-4179-b8e2-9fc07d847f5a" containerName="maas-api"
Jul 30 01:13:41.460334 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460160    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e6fe840-95ea-4f5d-be96-55751b512705" containerName="maas-api"
Jul 30 01:13:41.460334 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460165    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aa71380-9fb5-47a6-800b-4e6d1b81b862" containerName="revoke-keys"
Jul 30 01:13:41.460334 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460171    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="637a7f86-1d4e-419a-aea6-bdaff9d4044c" containerName="payload-processing"
Jul 30 01:13:41.460334 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460178    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ee3a598-1308-44de-9775-86a0b4bda898" containerName="payload-processing"
Jul 30 01:13:41.460334 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460184    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d582f6c-cdf2-4880-bae4-f3c523131855" containerName="maas-api"
Jul 30 01:13:41.460334 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460189    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f3ee87d-985d-4983-8609-c7feb4e2970d" containerName="payload-pre-processing"
Jul 30 01:13:41.460334 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.460206    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2fd9bdd-95c0-4ea0-bcdf-c32bfee1f12e" containerName="payload-pre-processing"
Jul 30 01:13:41.463309 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.463288    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p9qsq/must-gather-v9glv"
Jul 30 01:13:41.466849 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.466827    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p9qsq\"/\"kube-root-ca.crt\""
Jul 30 01:13:41.467216 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.467202    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p9qsq\"/\"openshift-service-ca.crt\""
Jul 30 01:13:41.473493 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.473466    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p9qsq\"/\"default-dockercfg-6smbp\""
Jul 30 01:13:41.475637 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.475611    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p9qsq/must-gather-v9glv"]
Jul 30 01:13:41.507302 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.507277    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3b5464a8-eb7e-4348-98b1-65e8841ca594-must-gather-output\") pod \"must-gather-v9glv\" (UID: \"3b5464a8-eb7e-4348-98b1-65e8841ca594\") " pod="openshift-must-gather-p9qsq/must-gather-v9glv"
Jul 30 01:13:41.507405 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.507342    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvq8p\" (UniqueName: \"kubernetes.io/projected/3b5464a8-eb7e-4348-98b1-65e8841ca594-kube-api-access-xvq8p\") pod \"must-gather-v9glv\" (UID: \"3b5464a8-eb7e-4348-98b1-65e8841ca594\") " pod="openshift-must-gather-p9qsq/must-gather-v9glv"
Jul 30 01:13:41.608381 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.608354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3b5464a8-eb7e-4348-98b1-65e8841ca594-must-gather-output\") pod \"must-gather-v9glv\" (UID: \"3b5464a8-eb7e-4348-98b1-65e8841ca594\") " pod="openshift-must-gather-p9qsq/must-gather-v9glv"
Jul 30 01:13:41.608492 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.608418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvq8p\" (UniqueName: \"kubernetes.io/projected/3b5464a8-eb7e-4348-98b1-65e8841ca594-kube-api-access-xvq8p\") pod \"must-gather-v9glv\" (UID: \"3b5464a8-eb7e-4348-98b1-65e8841ca594\") " pod="openshift-must-gather-p9qsq/must-gather-v9glv"
Jul 30 01:13:41.608687 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.608672    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3b5464a8-eb7e-4348-98b1-65e8841ca594-must-gather-output\") pod \"must-gather-v9glv\" (UID: \"3b5464a8-eb7e-4348-98b1-65e8841ca594\") " pod="openshift-must-gather-p9qsq/must-gather-v9glv"
Jul 30 01:13:41.617162 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.617134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvq8p\" (UniqueName: \"kubernetes.io/projected/3b5464a8-eb7e-4348-98b1-65e8841ca594-kube-api-access-xvq8p\") pod \"must-gather-v9glv\" (UID: \"3b5464a8-eb7e-4348-98b1-65e8841ca594\") " pod="openshift-must-gather-p9qsq/must-gather-v9glv"
Jul 30 01:13:41.773400 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.773336    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p9qsq/must-gather-v9glv"
Jul 30 01:13:41.912113 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:41.912089    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p9qsq/must-gather-v9glv"]
Jul 30 01:13:41.914046 ip-10-0-135-68 kubenswrapper[2579]: W0730 01:13:41.914015    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b5464a8_eb7e_4348_98b1_65e8841ca594.slice/crio-50c5426131944bdbe607dfa9c0d34070b36df6f94371a4183931ad271e307cbf WatchSource:0}: Error finding container 50c5426131944bdbe607dfa9c0d34070b36df6f94371a4183931ad271e307cbf: Status 404 returned error can't find the container with id 50c5426131944bdbe607dfa9c0d34070b36df6f94371a4183931ad271e307cbf
Jul 30 01:13:42.910126 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:42.910098    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p9qsq/must-gather-v9glv" event={"ID":"3b5464a8-eb7e-4348-98b1-65e8841ca594","Type":"ContainerStarted","Data":"50c5426131944bdbe607dfa9c0d34070b36df6f94371a4183931ad271e307cbf"}
Jul 30 01:13:43.920377 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:43.919644    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p9qsq/must-gather-v9glv" event={"ID":"3b5464a8-eb7e-4348-98b1-65e8841ca594","Type":"ContainerStarted","Data":"4bf610203a5f1551ff6bd73f2123700ae1cd1aefd9549ad21d15568164aee77a"}
Jul 30 01:13:43.920377 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:43.919707    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p9qsq/must-gather-v9glv" event={"ID":"3b5464a8-eb7e-4348-98b1-65e8841ca594","Type":"ContainerStarted","Data":"e54019a7ca70d1794af07d5b1fcc8f64cfe683ebb0262f0d17099c58a8383804"}
Jul 30 01:13:43.941657 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:43.941575    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p9qsq/must-gather-v9glv" podStartSLOduration=2.025209066 podStartE2EDuration="2.941554243s" totalImagesPullingTime="916.345177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:13:43.941303785 +0000 UTC m=+3349.238570718" watchObservedRunningTime="2026-07-30 01:13:43.941554243 +0000 UTC m=+3349.238821180"
Jul 30 01:13:44.476809 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:44.476778    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-86f4q_ebd8b209-a805-4976-bb8c-24cba1f01d86/global-pull-secret-syncer/0.log"
Jul 30 01:13:44.697783 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:44.697742    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qbkcd_182ce8f5-d95e-479e-9adf-a5571616b663/konnectivity-agent/0.log"
Jul 30 01:13:44.784225 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:44.784150    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-68.ec2.internal_260636c800f3b5abe8a8cc948b2e682e/haproxy/0.log"
Jul 30 01:13:49.365086 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:49.365058    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-h8mmx_7fbd644a-580f-44e0-89cf-121ac1eb0c80/kuadrant-console-plugin/0.log"
Jul 30 01:13:51.497819 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:51.497788    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f26fbc8a-a80a-4d45-996f-d26e043a8627/alertmanager/0.log"
Jul 30 01:13:51.526157 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:51.526131    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f26fbc8a-a80a-4d45-996f-d26e043a8627/config-reloader/0.log"
Jul 30 01:13:51.552187 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:51.552155    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f26fbc8a-a80a-4d45-996f-d26e043a8627/kube-rbac-proxy-web/0.log"
Jul 30 01:13:51.588014 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:51.587940    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f26fbc8a-a80a-4d45-996f-d26e043a8627/kube-rbac-proxy/0.log"
Jul 30 01:13:51.621043 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:51.621015    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f26fbc8a-a80a-4d45-996f-d26e043a8627/kube-rbac-proxy-metric/0.log"
Jul 30 01:13:51.648494 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:51.648469    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f26fbc8a-a80a-4d45-996f-d26e043a8627/prom-label-proxy/0.log"
Jul 30 01:13:51.679036 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:51.678974    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f26fbc8a-a80a-4d45-996f-d26e043a8627/init-config-reloader/0.log"
Jul 30 01:13:51.848131 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:51.847999    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-75c58d4475-nwtmf_0de6fb0f-5338-4568-a3b6-15a05b99b434/metrics-server/0.log"
Jul 30 01:13:51.881844 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:51.881773    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-44mvp_c9e278fb-935e-4397-8856-630640105426/monitoring-plugin/0.log"
Jul 30 01:13:52.107301 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:52.107194    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lf4z8_d7a41fca-3a9c-469a-ad8a-79047736d209/node-exporter/0.log"
Jul 30 01:13:52.129762 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:52.129736    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lf4z8_d7a41fca-3a9c-469a-ad8a-79047736d209/kube-rbac-proxy/0.log"
Jul 30 01:13:52.153788 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:52.153745    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lf4z8_d7a41fca-3a9c-469a-ad8a-79047736d209/init-textfile/0.log"
Jul 30 01:13:52.308083 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:52.308021    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_789934fb-707f-4b9d-a425-92cfe9c3b66a/prometheus/0.log"
Jul 30 01:13:52.349933 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:52.349890    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_789934fb-707f-4b9d-a425-92cfe9c3b66a/config-reloader/0.log"
Jul 30 01:13:52.375643 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:52.375609    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_789934fb-707f-4b9d-a425-92cfe9c3b66a/thanos-sidecar/0.log"
Jul 30 01:13:52.399074 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:52.399041    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_789934fb-707f-4b9d-a425-92cfe9c3b66a/kube-rbac-proxy-web/0.log"
Jul 30 01:13:52.422767 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:52.422736    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_789934fb-707f-4b9d-a425-92cfe9c3b66a/kube-rbac-proxy/0.log"
Jul 30 01:13:52.451593 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:52.451560    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_789934fb-707f-4b9d-a425-92cfe9c3b66a/kube-rbac-proxy-thanos/0.log"
Jul 30 01:13:52.486267 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:52.485656    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_789934fb-707f-4b9d-a425-92cfe9c3b66a/init-config-reloader/0.log"
Jul 30 01:13:52.534199 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:52.534172    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-7j8rn_28d39885-3cc1-4f91-80d1-b8d0cd31bef1/prometheus-operator/0.log"
Jul 30 01:13:52.557623 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:52.557553    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-7j8rn_28d39885-3cc1-4f91-80d1-b8d0cd31bef1/kube-rbac-proxy/0.log"
Jul 30 01:13:52.711648 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:52.711617    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b799ccf8b-62djb_3aa3fcdd-0019-4f8c-a799-6509d727d472/thanos-query/0.log"
Jul 30 01:13:52.738071 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:52.738036    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b799ccf8b-62djb_3aa3fcdd-0019-4f8c-a799-6509d727d472/kube-rbac-proxy-web/0.log"
Jul 30 01:13:52.762811 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:52.762772    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b799ccf8b-62djb_3aa3fcdd-0019-4f8c-a799-6509d727d472/kube-rbac-proxy/0.log"
Jul 30 01:13:52.786680 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:52.786651    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b799ccf8b-62djb_3aa3fcdd-0019-4f8c-a799-6509d727d472/prom-label-proxy/0.log"
Jul 30 01:13:52.811102 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:52.811076    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b799ccf8b-62djb_3aa3fcdd-0019-4f8c-a799-6509d727d472/kube-rbac-proxy-rules/0.log"
Jul 30 01:13:52.839100 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:52.839043    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b799ccf8b-62djb_3aa3fcdd-0019-4f8c-a799-6509d727d472/kube-rbac-proxy-metrics/0.log"
Jul 30 01:13:54.648954 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:54.648919    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"]
Jul 30 01:13:54.657682 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:54.657639    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"
Jul 30 01:13:54.675479 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:54.675451    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"]
Jul 30 01:13:54.763902 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:54.763544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8fb5d11-0fe3-4a3e-924f-3977a825652c-podres\") pod \"perf-node-gather-daemonset-lcbwv\" (UID: \"d8fb5d11-0fe3-4a3e-924f-3977a825652c\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"
Jul 30 01:13:54.763902 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:54.763633    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8fb5d11-0fe3-4a3e-924f-3977a825652c-lib-modules\") pod \"perf-node-gather-daemonset-lcbwv\" (UID: \"d8fb5d11-0fe3-4a3e-924f-3977a825652c\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"
Jul 30 01:13:54.763902 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:54.763685    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8fb5d11-0fe3-4a3e-924f-3977a825652c-sys\") pod \"perf-node-gather-daemonset-lcbwv\" (UID: \"d8fb5d11-0fe3-4a3e-924f-3977a825652c\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"
Jul 30 01:13:54.763902 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:54.763729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8fb5d11-0fe3-4a3e-924f-3977a825652c-proc\") pod \"perf-node-gather-daemonset-lcbwv\" (UID: \"d8fb5d11-0fe3-4a3e-924f-3977a825652c\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"
Jul 30 01:13:54.763902 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:54.763772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d28j2\" (UniqueName: \"kubernetes.io/projected/d8fb5d11-0fe3-4a3e-924f-3977a825652c-kube-api-access-d28j2\") pod \"perf-node-gather-daemonset-lcbwv\" (UID: \"d8fb5d11-0fe3-4a3e-924f-3977a825652c\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"
Jul 30 01:13:54.865120 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:54.865074    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8fb5d11-0fe3-4a3e-924f-3977a825652c-proc\") pod \"perf-node-gather-daemonset-lcbwv\" (UID: \"d8fb5d11-0fe3-4a3e-924f-3977a825652c\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"
Jul 30 01:13:54.865344 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:54.865157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d28j2\" (UniqueName: \"kubernetes.io/projected/d8fb5d11-0fe3-4a3e-924f-3977a825652c-kube-api-access-d28j2\") pod \"perf-node-gather-daemonset-lcbwv\" (UID: \"d8fb5d11-0fe3-4a3e-924f-3977a825652c\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"
Jul 30 01:13:54.865344 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:54.865261    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8fb5d11-0fe3-4a3e-924f-3977a825652c-podres\") pod \"perf-node-gather-daemonset-lcbwv\" (UID: \"d8fb5d11-0fe3-4a3e-924f-3977a825652c\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"
Jul 30 01:13:54.865344 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:54.865336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8fb5d11-0fe3-4a3e-924f-3977a825652c-lib-modules\") pod \"perf-node-gather-daemonset-lcbwv\" (UID: \"d8fb5d11-0fe3-4a3e-924f-3977a825652c\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"
Jul 30 01:13:54.865540 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:54.865380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8fb5d11-0fe3-4a3e-924f-3977a825652c-sys\") pod \"perf-node-gather-daemonset-lcbwv\" (UID: \"d8fb5d11-0fe3-4a3e-924f-3977a825652c\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"
Jul 30 01:13:54.865540 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:54.865500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8fb5d11-0fe3-4a3e-924f-3977a825652c-sys\") pod \"perf-node-gather-daemonset-lcbwv\" (UID: \"d8fb5d11-0fe3-4a3e-924f-3977a825652c\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"
Jul 30 01:13:54.865654 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:54.865633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8fb5d11-0fe3-4a3e-924f-3977a825652c-lib-modules\") pod \"perf-node-gather-daemonset-lcbwv\" (UID: \"d8fb5d11-0fe3-4a3e-924f-3977a825652c\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"
Jul 30 01:13:54.865654 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:54.865634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8fb5d11-0fe3-4a3e-924f-3977a825652c-podres\") pod \"perf-node-gather-daemonset-lcbwv\" (UID: \"d8fb5d11-0fe3-4a3e-924f-3977a825652c\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"
Jul 30 01:13:54.865773 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:54.865693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8fb5d11-0fe3-4a3e-924f-3977a825652c-proc\") pod \"perf-node-gather-daemonset-lcbwv\" (UID: \"d8fb5d11-0fe3-4a3e-924f-3977a825652c\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"
Jul 30 01:13:54.876323 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:54.876294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d28j2\" (UniqueName: \"kubernetes.io/projected/d8fb5d11-0fe3-4a3e-924f-3977a825652c-kube-api-access-d28j2\") pod \"perf-node-gather-daemonset-lcbwv\" (UID: \"d8fb5d11-0fe3-4a3e-924f-3977a825652c\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"
Jul 30 01:13:54.972398 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:54.972308    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"
Jul 30 01:13:55.144477 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:55.144219    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"]
Jul 30 01:13:55.145817 ip-10-0-135-68 kubenswrapper[2579]: W0730 01:13:55.145763    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd8fb5d11_0fe3_4a3e_924f_3977a825652c.slice/crio-d2df6ecc694e6ca2cca44c29055191cedef2db018449a5a47e970636a4a55053 WatchSource:0}: Error finding container d2df6ecc694e6ca2cca44c29055191cedef2db018449a5a47e970636a4a55053: Status 404 returned error can't find the container with id d2df6ecc694e6ca2cca44c29055191cedef2db018449a5a47e970636a4a55053
Jul 30 01:13:55.431319 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:55.431285    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-b5764f55-52b46_b64488b7-69fd-400d-8028-7d6df50c4d45/console/0.log"
Jul 30 01:13:56.009648 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:56.009603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv" event={"ID":"d8fb5d11-0fe3-4a3e-924f-3977a825652c","Type":"ContainerStarted","Data":"8d23f07e74b296458d5d0147d3bf17c1ed4f60f541e0a34644e23f2a9b7f9454"}
Jul 30 01:13:56.010049 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:56.009663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv" event={"ID":"d8fb5d11-0fe3-4a3e-924f-3977a825652c","Type":"ContainerStarted","Data":"d2df6ecc694e6ca2cca44c29055191cedef2db018449a5a47e970636a4a55053"}
Jul 30 01:13:56.010049 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:56.009720    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"
Jul 30 01:13:56.027798 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:56.027745    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv" podStartSLOduration=2.027728365 podStartE2EDuration="2.027728365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 01:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:13:56.025461705 +0000 UTC m=+3361.322728637" watchObservedRunningTime="2026-07-30 01:13:56.027728365 +0000 UTC m=+3361.324995297"
Jul 30 01:13:57.257939 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:57.257911    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lcr97_b39fd448-9f15-4275-86cf-5d233023c5a5/dns/0.log"
Jul 30 01:13:57.280493 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:57.280461    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lcr97_b39fd448-9f15-4275-86cf-5d233023c5a5/kube-rbac-proxy/0.log"
Jul 30 01:13:57.308920 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:57.308894    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2jclk_318afb78-6a56-4524-8c63-8a3c2b223271/dns-node-resolver/0.log"
Jul 30 01:13:57.988418 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:57.988369    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pwx6g_836811a6-7584-4818-884a-59ce980afe22/node-ca/0.log"
Jul 30 01:13:59.190960 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:59.190923    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-94d649dc8-lm6p2_fb873962-c767-48e6-b6b3-09435499efa5/kube-auth-proxy/0.log"
Jul 30 01:13:59.287389 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:59.287359    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-m7n6j_9fc56b79-4db8-4489-a8c5-95e6349ce60d/payload-pre-processing/0.log"
Jul 30 01:13:59.325119 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:13:59.325086    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9k8fv_2e048fda-d493-49d2-955e-a4f1bb2a7f8e/payload-processing/0.log"
Jul 30 01:14:00.208456 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:14:00.208428    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7pwht_2c0002be-c29e-4574-9d66-739390a6a6d3/serve-healthcheck-canary/0.log"
Jul 30 01:14:00.861169 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:14:00.861139    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9v8g2_466f60c6-0f78-4017-a90f-87489f71021c/kube-rbac-proxy/0.log"
Jul 30 01:14:00.895266 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:14:00.895221    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9v8g2_466f60c6-0f78-4017-a90f-87489f71021c/exporter/0.log"
Jul 30 01:14:00.931639 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:14:00.931555    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9v8g2_466f60c6-0f78-4017-a90f-87489f71021c/extractor/0.log"
Jul 30 01:14:02.028340 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:14:02.028311    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-lcbwv"
Jul 30 01:14:03.841297 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:14:03.841263    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bd495bbc5-jvmkl_6440c0bc-07f7-4dfd-af29-c014028ecd2a/manager/0.log"
Jul 30 01:14:04.232405 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:14:04.232373    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b74bc5486-z878b_27f4f29e-3606-46f6-aaa6-6bae1807ace8/manager/0.log"
Jul 30 01:14:04.253167 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:14:04.253137    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b74bc5486-z878b_27f4f29e-3606-46f6-aaa6-6bae1807ace8/copy-manifests/0.log"
Jul 30 01:14:05.276446 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:14:05.276415    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-94d649dc8-lm6p2_fb873962-c767-48e6-b6b3-09435499efa5/kube-auth-proxy/0.log"
Jul 30 01:14:05.347928 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:14:05.347830    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-m7n6j_9fc56b79-4db8-4489-a8c5-95e6349ce60d/payload-pre-processing/0.log"
Jul 30 01:14:05.371017 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:14:05.370991    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9k8fv_2e048fda-d493-49d2-955e-a4f1bb2a7f8e/payload-processing/0.log"
Jul 30 01:14:05.614312 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:14:05.614208    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-685c7855bf-g9j9l_178cfbc9-70ed-48ac-ad08-c04cabcb55db/manager/0.log"
Jul 30 01:14:05.694024 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:14:05.693996    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-knr7k_a21fa613-f28c-489c-89cd-337991e65557/openshift-lws-operator/0.log"
Jul 30 01:14:07.051446 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:14:07.051416    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-94d649dc8-lm6p2_fb873962-c767-48e6-b6b3-09435499efa5/kube-auth-proxy/0.log"
Jul 30 01:14:07.127746 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:14:07.127715    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-m7n6j_9fc56b79-4db8-4489-a8c5-95e6349ce60d/payload-pre-processing/0.log"
Jul 30 01:14:07.151258 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:14:07.151222    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9k8fv_2e048fda-d493-49d2-955e-a4f1bb2a7f8e/payload-processing/0.log"
Jul 30 01:14:12.220126 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:14:12.220098    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9bqqn_7471b635-17bb-4f3a-b41f-ee9fa0f9871e/kube-multus/0.log"
Jul 30 01:14:12.468929 ip-10-0-135-68 kubenswrapper[2579]: I0730 01:14:12.468901    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jfkf9_7f6d919d-802a-4eb8-a7ec-948626dfc26b/kube-multus-additional-cni-plugins/0.log"