Aug 04 03:09:29.201287 ip-10-0-138-99 systemd[1]: Starting Kubernetes Kubelet...
Aug 04 03:09:29.603220 ip-10-0-138-99 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 03:09:29.603220 ip-10-0-138-99 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Aug 04 03:09:29.603220 ip-10-0-138-99 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 03:09:29.603220 ip-10-0-138-99 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Aug 04 03:09:29.603220 ip-10-0-138-99 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 03:09:29.605149 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.604976    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Aug 04 03:09:29.610539 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610516    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 03:09:29.610539 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610534    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 03:09:29.610539 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610538    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 03:09:29.610539 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610541    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 03:09:29.610539 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610544    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 03:09:29.610539 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610547    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 03:09:29.610752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610550    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 03:09:29.610752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610553    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 03:09:29.610752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610556    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 03:09:29.610752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610558    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 03:09:29.610752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610561    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 03:09:29.610752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610564    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 03:09:29.610752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610566    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 03:09:29.610752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610569    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 03:09:29.610752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610571    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 03:09:29.610752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610575    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 03:09:29.610752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610577    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 03:09:29.610752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610580    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 03:09:29.610752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610583    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 03:09:29.610752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610586    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 03:09:29.610752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610588    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 03:09:29.610752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610598    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 03:09:29.610752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610601    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 03:09:29.610752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610603    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 03:09:29.610752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610606    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 03:09:29.611209 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610608    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 03:09:29.611209 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610611    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 03:09:29.611209 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610614    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 03:09:29.611209 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610616    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 03:09:29.611209 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610619    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 03:09:29.611209 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610622    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 03:09:29.611209 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610624    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 03:09:29.611209 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610627    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 03:09:29.611209 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610629    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 03:09:29.611209 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610633    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 03:09:29.611209 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610638    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 03:09:29.611209 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610640    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 03:09:29.611209 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610643    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 03:09:29.611209 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610646    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 03:09:29.611209 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610650    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 03:09:29.611209 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610653    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 03:09:29.611209 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610656    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 03:09:29.611209 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610659    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 03:09:29.611209 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610662    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610665    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610667    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610670    2579 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610673    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610675    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610678    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610680    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610683    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610685    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610688    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610691    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610694    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610696    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610699    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610701    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610704    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610706    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610709    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610711    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 03:09:29.611703 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610716    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610720    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610722    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610725    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610727    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610730    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610733    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610736    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610739    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610742    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610744    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610747    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610751    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610753    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610756    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610759    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610761    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610764    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610766    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610769    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 03:09:29.612198 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610772    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610775    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610777    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610780    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610783    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610786    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610789    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610791    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610795    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610799    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610802    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610805    2579 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610808    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610811    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610814    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.610816    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611548    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611555    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611558    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611561    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 03:09:29.612679 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611565    2579 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 03:09:29.613181 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611568    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 03:09:29.613181 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611571    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 03:09:29.613181 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611574    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 03:09:29.613181 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611578    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 03:09:29.613181 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611582    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 03:09:29.613181 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611585    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 03:09:29.613181 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611587    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 03:09:29.613181 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611590    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 03:09:29.613181 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611593    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 03:09:29.613181 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611595    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 03:09:29.613181 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611598    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 03:09:29.613181 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611601    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 03:09:29.613181 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611604    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 03:09:29.613181 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611607    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 03:09:29.613181 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611609    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 03:09:29.613181 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611612    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 03:09:29.613181 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611614    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 03:09:29.613181 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611617    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 03:09:29.613181 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611619    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611622    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611625    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611627    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611630    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611632    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611635    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611637    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611640    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611642    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611645    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611647    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611651    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611655    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611657    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611661    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611663    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611666    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611669    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611672    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 03:09:29.613659 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611674    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611676    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611679    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611681    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611684    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611687    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611689    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611693    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611695    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611698    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611700    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611703    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611705    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611708    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611711    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611714    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611716    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611719    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611722    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611724    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 03:09:29.614189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611727    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611729    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611732    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611735    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611738    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611740    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611743    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611747    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611750    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611753    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611756    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611758    2579 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611762    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611765    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611767    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611770    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611772    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611775    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611777    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611780    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 03:09:29.614675 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611783    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 03:09:29.615189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611786    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 03:09:29.615189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611788    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 03:09:29.615189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611791    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 03:09:29.615189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611793    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 03:09:29.615189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611796    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 03:09:29.615189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611800    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 03:09:29.615189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611803    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 03:09:29.615189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611807    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 03:09:29.615189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611809    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 03:09:29.615189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611813    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 03:09:29.615189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611816    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 03:09:29.615189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611819    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 03:09:29.615189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611822    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 03:09:29.615189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611825    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 03:09:29.615189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.611828    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 03:09:29.615189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612254    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 03:09:29.615189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612260    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 03:09:29.615189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612263    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 03:09:29.615189 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612266    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612270    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612273    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612276    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612279    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612281    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612284    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612286    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612289    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612292    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612294    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612297    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612299    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612303    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612305    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612308    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612311    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612314    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612317    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612320    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 03:09:29.615644 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612322    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612325    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612327    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612330    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612332    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612335    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612338    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612342    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612345    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612347    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612350    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612352    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612355    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612357    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612360    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612362    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612365    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612367    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612370    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612373    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 03:09:29.616205 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612375    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612377    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612380    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612382    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612385    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612388    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612391    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612393    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612396    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612398    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612401    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612403    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612406    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612408    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612411    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612414    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612416    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612418    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612422    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612424    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 03:09:29.616684 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612427    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612430    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612432    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612435    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612437    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612439    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612442    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612445    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612447    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612450    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612452    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612455    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612459    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612462    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612465    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612468    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612473    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612475    2579 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612478    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612482    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 03:09:29.617195 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612485    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612488    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612490    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612494    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612497    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612500    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612503    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612505    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612508    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612510    2579 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612513    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612516    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612518    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612521    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612523    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.612527    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612596    2579 flags.go:64] FLAG: --address="0.0.0.0"
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612603    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612610    2579 flags.go:64] FLAG: --anonymous-auth="true"
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612614    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612618    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Aug 04 03:09:29.617686 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612621    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612625    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612634    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612637    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612639    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612643    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612646    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612650    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612654    2579 flags.go:64] FLAG: --cgroup-root=""
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612656    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612659    2579 flags.go:64] FLAG: --client-ca-file=""
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612663    2579 flags.go:64] FLAG: --cloud-provider="external"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612666    2579 flags.go:64] FLAG: --cluster-dns="[]"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612670    2579 flags.go:64] FLAG: --cluster-domain=""
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612673    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612676    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612679    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612682    2579 flags.go:64] FLAG: --container-log-max-files="5"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612687    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612690    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612693    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612696    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612699    2579 flags.go:64] FLAG: --contention-profiling="false"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612702    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Aug 04 03:09:29.618230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612705    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612709    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612711    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612716    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612719    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612721    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612724    2579 flags.go:64] FLAG: --enable-load-reader="false"
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612727    2579 flags.go:64] FLAG: --enable-server="true"
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612730    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612735    2579 flags.go:64] FLAG: --event-burst="100"
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612738    2579 flags.go:64] FLAG: --event-qps="50"
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612741    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612744    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612747    2579 flags.go:64] FLAG: --eviction-hard=""
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612750    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612755    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612758    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612761    2579 flags.go:64] FLAG: --eviction-soft=""
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612764    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612767    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612777    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612780    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612783    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612786    2579 flags.go:64] FLAG: --fail-swap-on="true"
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612788    2579 flags.go:64] FLAG: --feature-gates=""
Aug 04 03:09:29.619242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612796    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612800    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612803    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612806    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612809    2579 flags.go:64] FLAG: --healthz-port="10248"
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612812    2579 flags.go:64] FLAG: --help="false"
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612815    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-138-99.ec2.internal"
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612818    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612821    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612825    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612828    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612831    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612834    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612837    2579 flags.go:64] FLAG: --image-service-endpoint=""
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612840    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612843    2579 flags.go:64] FLAG: --kube-api-burst="100"
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612846    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612849    2579 flags.go:64] FLAG: --kube-api-qps="50"
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612852    2579 flags.go:64] FLAG: --kube-reserved=""
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612855    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612858    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612861    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612865    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612868    2579 flags.go:64] FLAG: --lock-file=""
Aug 04 03:09:29.619958 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612871    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612874    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612877    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612882    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612886    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612889    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612892    2579 flags.go:64] FLAG: --logging-format="text"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612895    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612898    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612901    2579 flags.go:64] FLAG: --manifest-url=""
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612904    2579 flags.go:64] FLAG: --manifest-url-header=""
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612908    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612911    2579 flags.go:64] FLAG: --max-open-files="1000000"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612915    2579 flags.go:64] FLAG: --max-pods="110"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612918    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612921    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612924    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612927    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612930    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612933    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612936    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612943    2579 flags.go:64] FLAG: --node-status-max-images="50"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612947    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612950    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Aug 04 03:09:29.620639 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612952    2579 flags.go:64] FLAG: --pod-cidr=""
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612968    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612973    2579 flags.go:64] FLAG: --pod-manifest-path=""
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612977    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612980    2579 flags.go:64] FLAG: --pods-per-core="0"
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612983    2579 flags.go:64] FLAG: --port="10250"
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612987    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612990    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-007148792454eb94d"
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612993    2579 flags.go:64] FLAG: --qos-reserved=""
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.612999    2579 flags.go:64] FLAG: --read-only-port="10255"
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613002    2579 flags.go:64] FLAG: --register-node="true"
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613005    2579 flags.go:64] FLAG: --register-with-taints=""
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613008    2579 flags.go:64] FLAG: --registry-burst="10"
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613012    2579 flags.go:64] FLAG: --registry-qps="5"
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613015    2579 flags.go:64] FLAG: --reserved-cpus=""
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613018    2579 flags.go:64] FLAG: --reserved-memory=""
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613029    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613032    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613035    2579 flags.go:64] FLAG: --rotate-certificates="false"
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613038    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613041    2579 flags.go:64] FLAG: --runonce="false"
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613045    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613048    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613051    2579 flags.go:64] FLAG: --seccomp-default="false"
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613053    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Aug 04 03:09:29.621296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613056    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613059    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613062    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613065    2579 flags.go:64] FLAG: --storage-driver-password="root"
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613068    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613071    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613074    2579 flags.go:64] FLAG: --storage-driver-user="root"
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613077    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613080    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613083    2579 flags.go:64] FLAG: --system-cgroups=""
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613085    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613091    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613094    2579 flags.go:64] FLAG: --tls-cert-file=""
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613099    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613103    2579 flags.go:64] FLAG: --tls-min-version=""
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613106    2579 flags.go:64] FLAG: --tls-private-key-file=""
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613110    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613113    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613116    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613119    2579 flags.go:64] FLAG: --v="2"
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613123    2579 flags.go:64] FLAG: --version="false"
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613127    2579 flags.go:64] FLAG: --vmodule=""
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613131    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613134    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613258    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 03:09:29.621960 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613262    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613265    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613268    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613271    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613274    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613277    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613279    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613282    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613285    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613287    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613290    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613292    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613295    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613298    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613300    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613303    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613305    2579 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613308    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613311    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613313    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 03:09:29.622690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613318    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 03:09:29.623228 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613320    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 03:09:29.623228 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613323    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 03:09:29.623228 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613326    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 03:09:29.623228 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613329    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 03:09:29.623228 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613331    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 03:09:29.623228 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613334    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 03:09:29.623228 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613337    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 03:09:29.623228 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613339    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 03:09:29.623228 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613342    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 03:09:29.623228 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613345    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 03:09:29.623228 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613356    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 03:09:29.623228 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613359    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 03:09:29.623228 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613361    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 03:09:29.623228 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613364    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 03:09:29.623228 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613367    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 03:09:29.623228 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613369    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 03:09:29.623228 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613372    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 03:09:29.623228 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613375    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 03:09:29.623228 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613378    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 03:09:29.623766 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613382    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 03:09:29.623766 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613386    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 03:09:29.623766 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613390    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 03:09:29.623766 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613393    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 03:09:29.623766 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613396    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 03:09:29.623766 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613399    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 03:09:29.623766 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613403    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 03:09:29.623766 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613405    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 03:09:29.623766 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613408    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 03:09:29.623766 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613410    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 03:09:29.623766 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613413    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 03:09:29.623766 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613415    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 03:09:29.623766 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613419    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 03:09:29.623766 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613422    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 03:09:29.623766 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613425    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 03:09:29.623766 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613429    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 03:09:29.623766 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613431    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 03:09:29.623766 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613434    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 03:09:29.623766 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613437    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613439    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613442    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613444    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613449    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613451    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613453    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613456    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613458    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613461    2579 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613463    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613466    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613469    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613472    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613474    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613477    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613479    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613482    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613485    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613489    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 03:09:29.624332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613492    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 03:09:29.624870 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613494    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 03:09:29.624870 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613497    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 03:09:29.624870 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613500    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 03:09:29.624870 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613502    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 03:09:29.624870 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613505    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 03:09:29.624870 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613509    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 03:09:29.624870 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613512    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 03:09:29.624870 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613515    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 03:09:29.624870 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613519    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 03:09:29.624870 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613522    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 03:09:29.624870 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613524    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 03:09:29.624870 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613527    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 03:09:29.624870 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613530    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 03:09:29.624870 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613532    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 03:09:29.624870 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613535    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 03:09:29.624870 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613537    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 03:09:29.624870 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613541    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 03:09:29.624870 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613543    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 03:09:29.624870 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.613546    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 03:09:29.625391 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.613550    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 03:09:29.625391 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.620400    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Aug 04 03:09:29.625391 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.620418    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Aug 04 03:09:29.625391 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.620439    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Aug 04 03:09:29.625391 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.620445    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Aug 04 03:09:29.625391 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620507    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 03:09:29.625391 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620513    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 03:09:29.625391 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620518    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 03:09:29.625391 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620521    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 03:09:29.625391 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620524    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 03:09:29.625391 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620527    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 03:09:29.625391 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620530    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 03:09:29.625391 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620533    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 03:09:29.625391 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620536    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 03:09:29.625391 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620539    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 03:09:29.625800 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620542    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 03:09:29.625800 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620545    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 03:09:29.625800 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620548    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 03:09:29.625800 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620551    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 03:09:29.625800 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620554    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 03:09:29.625800 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620557    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 03:09:29.625800 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620560    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 03:09:29.625800 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620564    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 03:09:29.625800 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620568    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 03:09:29.625800 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620571    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 03:09:29.625800 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620574    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 03:09:29.625800 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620577    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 03:09:29.625800 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620579    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 03:09:29.625800 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620582    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 03:09:29.625800 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620588    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 03:09:29.625800 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620592    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 03:09:29.625800 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620595    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 03:09:29.625800 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620599    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 03:09:29.625800 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620602    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 03:09:29.626394 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620605    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 03:09:29.626394 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620608    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 03:09:29.626394 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620612    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 03:09:29.626394 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620616    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 03:09:29.626394 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620620    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 03:09:29.626394 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620623    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 03:09:29.626394 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620626    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 03:09:29.626394 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620630    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 03:09:29.626394 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620633    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 03:09:29.626394 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620636    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 03:09:29.626394 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620639    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 03:09:29.626394 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620642    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 03:09:29.626394 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620645    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 03:09:29.626394 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620648    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 03:09:29.626394 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620651    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 03:09:29.626394 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620654    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 03:09:29.626394 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620657    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 03:09:29.626394 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620661    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 03:09:29.626394 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620664    2579 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620666    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620669    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620672    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620675    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620677    2579 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620680    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620683    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620686    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620688    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620692    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620695    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620698    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620701    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620704    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620707    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620710    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620713    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620717    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620719    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 03:09:29.626897 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620722    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620725    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620728    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620730    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620733    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620736    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620739    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620741    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620745    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620747    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620750    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620753    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620756    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620758    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620761    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620764    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620767    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620770    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620773    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620775    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 03:09:29.627478 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620778    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 03:09:29.628046 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620781    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 03:09:29.628046 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620784    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 03:09:29.628046 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620787    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 03:09:29.628046 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620790    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 03:09:29.628046 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620793    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 03:09:29.628046 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620796    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 03:09:29.628046 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620799    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 03:09:29.628046 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620802    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 03:09:29.628046 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620805    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 03:09:29.628046 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620808    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 03:09:29.628046 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620811    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 03:09:29.628046 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.620818    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 03:09:29.628046 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620953    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 03:09:29.628046 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620958    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 03:09:29.628046 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620962    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620965    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620968    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620970    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620973    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620976    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620978    2579 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620981    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620984    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620986    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620989    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620992    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620994    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.620997    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621000    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621002    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621005    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621007    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621010    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621013    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 03:09:29.628447 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621015    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621018    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621021    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621024    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621027    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621030    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621045    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621048    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621051    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621054    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621057    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621060    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621063    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621066    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621069    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621072    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621074    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621077    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621080    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621082    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 03:09:29.628974 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621085    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 03:09:29.629525 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621088    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 03:09:29.629525 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621091    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 03:09:29.629525 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621094    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 03:09:29.629525 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621096    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 03:09:29.629525 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621099    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 03:09:29.629525 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621102    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 03:09:29.629525 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621105    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 03:09:29.629525 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621107    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 03:09:29.629525 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621110    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 03:09:29.629525 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621113    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 03:09:29.629525 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621115    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 03:09:29.629525 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621118    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 03:09:29.629525 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621121    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 03:09:29.629525 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621125    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 03:09:29.629525 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621128    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 03:09:29.629525 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621130    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 03:09:29.629525 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621133    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 03:09:29.629525 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621136    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 03:09:29.629525 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621140    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 03:09:29.630028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621144    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 03:09:29.630028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621146    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 03:09:29.630028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621149    2579 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 03:09:29.630028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621152    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 03:09:29.630028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621155    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 03:09:29.630028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621158    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 03:09:29.630028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621160    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 03:09:29.630028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621164    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 03:09:29.630028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621168    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 03:09:29.630028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621171    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 03:09:29.630028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621174    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 03:09:29.630028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621177    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 03:09:29.630028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621180    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 03:09:29.630028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621183    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 03:09:29.630028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621186    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 03:09:29.630028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621189    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 03:09:29.630028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621191    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 03:09:29.630028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621194    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 03:09:29.630028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621197    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 03:09:29.630722 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621199    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 03:09:29.630722 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621202    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 03:09:29.630722 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621205    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 03:09:29.630722 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621208    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 03:09:29.630722 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621211    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 03:09:29.630722 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621214    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 03:09:29.630722 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621217    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 03:09:29.630722 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621220    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 03:09:29.630722 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621224    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 03:09:29.630722 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621226    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 03:09:29.630722 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621230    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 03:09:29.630722 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621234    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 03:09:29.630722 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621237    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 03:09:29.630722 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621240    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 03:09:29.630722 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621243    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 03:09:29.630722 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621246    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 03:09:29.630722 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621249    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 03:09:29.630722 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:29.621252    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 03:09:29.631347 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.621257    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 03:09:29.631347 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.621363    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Aug 04 03:09:29.631347 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.623434    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Aug 04 03:09:29.631347 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.624683    2579 server.go:1019] "Starting client certificate rotation"
Aug 04 03:09:29.631347 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.624787    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 03:09:29.631347 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.624831    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 03:09:29.631347 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.627449    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 03:09:29.631347 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.629343    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 03:09:29.639677 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.639656    2579 log.go:25] "Validated CRI v1 runtime API"
Aug 04 03:09:29.641774 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.641759    2579 log.go:25] "Validated CRI v1 image API"
Aug 04 03:09:29.642728 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.642709    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Aug 04 03:09:29.645509 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.645488    2579 fs.go:135] Filesystem UUIDs: map[3d9fa0c8-754c-449c-afa3-ecb42e420987:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d1d75535-3177-4c40-9118-00ec8021bcf1:/dev/nvme0n1p4]
Aug 04 03:09:29.645584 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.645509    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Aug 04 03:09:29.651384 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.651263    2579 manager.go:216] Machine: {Timestamp:2026-08-04 03:09:29.649806588 +0000 UTC m=+0.329593722 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100927 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20a33cb1eb02d09d8ca73e4c9ab6a3 SystemUUID:ec20a33c-b1eb-02d0-9d8c-a73e4c9ab6a3 BootID:a1c3e04d-041c-4ba7-a44d-da4f4cfab28c Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e3:97:48:cc:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e3:97:48:cc:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:ee:16:3b:91:11 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Aug 04 03:09:29.651384 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.651381    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Aug 04 03:09:29.651517 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.651467    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Aug 04 03:09:29.654301 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.654277    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Aug 04 03:09:29.654449 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.654304    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-99.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Aug 04 03:09:29.655232 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.655214    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 03:09:29.655490 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.655478    2579 topology_manager.go:138] "Creating topology manager with none policy"
Aug 04 03:09:29.655533 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.655493    2579 container_manager_linux.go:309] "Creating device plugin manager"
Aug 04 03:09:29.655533 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.655502    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 03:09:29.656249 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.656238    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 03:09:29.657308 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.657296    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Aug 04 03:09:29.661676 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.661664    2579 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 03:09:29.661835 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.661824    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Aug 04 03:09:29.661876 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.661855    2579 kubelet.go:486] "Attempting to sync node with API server"
Aug 04 03:09:29.661876 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.661865    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Aug 04 03:09:29.661943 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.661877    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Aug 04 03:09:29.661943 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.661887    2579 kubelet.go:398] "Adding apiserver pod source"
Aug 04 03:09:29.661943 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.661895    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Aug 04 03:09:29.662290 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.662278    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 03:09:29.662327 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.662297    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 03:09:29.665706 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.665685    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Aug 04 03:09:29.667078 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.667064    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Aug 04 03:09:29.667135 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.667096    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Aug 04 03:09:29.668748 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.668732    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Aug 04 03:09:29.668748 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.668750    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Aug 04 03:09:29.668847 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.668757    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Aug 04 03:09:29.668847 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.668763    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Aug 04 03:09:29.668847 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.668772    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Aug 04 03:09:29.668847 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.668778    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Aug 04 03:09:29.668847 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.668786    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Aug 04 03:09:29.668847 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.668792    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Aug 04 03:09:29.668847 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.668799    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Aug 04 03:09:29.668847 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.668805    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Aug 04 03:09:29.668847 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.668814    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Aug 04 03:09:29.668847 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.668820    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Aug 04 03:09:29.669656 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.669642    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Aug 04 03:09:29.669656 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.669653    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Aug 04 03:09:29.672993 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.672977    2579 server.go:1268] "Started kubelet"
Aug 04 03:09:29.673127 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.673096    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Aug 04 03:09:29.673226 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.673211    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Aug 04 03:09:29.673365 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.673296    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Aug 04 03:09:29.673712 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.673690    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Aug 04 03:09:29.673838 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:29.673689    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Aug 04 03:09:29.673838 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:29.673804    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Aug 04 03:09:29.673904 ip-10-0-138-99 systemd[1]: Started Kubernetes Kubelet.
Aug 04 03:09:29.674745 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.674524    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Aug 04 03:09:29.679660 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.679603    2579 server.go:310] "Adding debug handlers to kubelet server"
Aug 04 03:09:29.680652 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:29.680627    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Aug 04 03:09:29.681100 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.681086    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Aug 04 03:09:29.681192 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.681101    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Aug 04 03:09:29.681192 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:29.681181    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Aug 04 03:09:29.681735 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.681709    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Aug 04 03:09:29.681735 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.681730    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Aug 04 03:09:29.681884 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.681717    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Aug 04 03:09:29.681884 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.681868    2579 reconstruct.go:97] "Volume reconstruction finished"
Aug 04 03:09:29.681884 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.681879    2579 reconciler.go:29] "Reconciler: start to sync state"
Aug 04 03:09:29.682064 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.681989    2579 factory.go:55] Registering systemd factory
Aug 04 03:09:29.682064 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.682059    2579 factory.go:223] Registration of the systemd container factory successfully
Aug 04 03:09:29.682351 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.682331    2579 factory.go:153] Registering CRI-O factory
Aug 04 03:09:29.682351 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.682353    2579 factory.go:223] Registration of the crio container factory successfully
Aug 04 03:09:29.682474 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.682410    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Aug 04 03:09:29.682474 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.682435    2579 factory.go:103] Registering Raw factory
Aug 04 03:09:29.682474 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.682452    2579 manager.go:1196] Started watching for new ooms in manager
Aug 04 03:09:29.683181 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.683166    2579 manager.go:349] Starting recovery of all containers
Aug 04 03:09:29.685672 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:29.682784    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-99.ec2.internal.18c87b6f2ca441a3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-99.ec2.internal,UID:ip-10-0-138-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-99.ec2.internal,},FirstTimestamp:2026-08-04 03:09:29.672950179 +0000 UTC m=+0.352737291,LastTimestamp:2026-08-04 03:09:29.672950179 +0000 UTC m=+0.352737291,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-99.ec2.internal,}"
Aug 04 03:09:29.691283 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:29.691257    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Aug 04 03:09:29.691375 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:29.691329    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Aug 04 03:09:29.694667 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.694650    2579 manager.go:354] Recovery completed
Aug 04 03:09:29.696777 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:29.696601    2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Aug 04 03:09:29.700654 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.700641    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 03:09:29.700870 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.700853    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jp2w6"
Aug 04 03:09:29.702714 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.702696    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 03:09:29.702798 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.702734    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 03:09:29.702798 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.702754    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientPID"
Aug 04 03:09:29.703390 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.703375    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Aug 04 03:09:29.703390 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.703388    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Aug 04 03:09:29.703548 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.703397    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 03:09:29.703548 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.703408    2579 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 03:09:29.705361 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:29.705243    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-99.ec2.internal.18c87b6f2e6a721b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-99.ec2.internal,UID:ip-10-0-138-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-99.ec2.internal,},FirstTimestamp:2026-08-04 03:09:29.702715931 +0000 UTC m=+0.382503047,LastTimestamp:2026-08-04 03:09:29.702715931 +0000 UTC m=+0.382503047,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-99.ec2.internal,}"
Aug 04 03:09:29.705811 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.705794    2579 policy_none.go:49] "None policy: Start"
Aug 04 03:09:29.705921 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.705911    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Aug 04 03:09:29.706035 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.706023    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Aug 04 03:09:29.706755 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.706728    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 03:09:29.706843 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.706767    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 03:09:29.706843 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.706786    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientPID"
Aug 04 03:09:29.708372 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.708355    2579 policy_none.go:47] "Start"
Aug 04 03:09:29.708940 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.708923    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jp2w6"
Aug 04 03:09:29.747726 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.747704    2579 manager.go:337] "Starting Device Plugin manager"
Aug 04 03:09:29.747867 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:29.747751    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Aug 04 03:09:29.747867 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.747765    2579 server.go:87] "Starting device plugin registration server"
Aug 04 03:09:29.748138 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.748124    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Aug 04 03:09:29.748178 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.748143    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Aug 04 03:09:29.748269 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.748250    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Aug 04 03:09:29.748365 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.748348    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Aug 04 03:09:29.748418 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.748368    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Aug 04 03:09:29.748985 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:29.748961    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Aug 04 03:09:29.749095 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:29.749030    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-99.ec2.internal\" not found"
Aug 04 03:09:29.836981 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.836927    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Aug 04 03:09:29.839458 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.838252    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Aug 04 03:09:29.839458 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.838305    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Aug 04 03:09:29.839458 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.838329    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Aug 04 03:09:29.839458 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:29.838371    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Aug 04 03:09:29.840926 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.840907    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 03:09:29.848905 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.848888    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 03:09:29.849795 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.849779    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 03:09:29.849877 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.849809    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 03:09:29.849877 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.849822    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientPID"
Aug 04 03:09:29.849877 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.849846    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-99.ec2.internal"
Aug 04 03:09:29.858743 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.858685    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-99.ec2.internal"
Aug 04 03:09:29.858743 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:29.858714    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-99.ec2.internal\": node \"ip-10-0-138-99.ec2.internal\" not found"
Aug 04 03:09:29.886236 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:29.886209    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Aug 04 03:09:29.938473 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.938436    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal"]
Aug 04 03:09:29.938609 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.938523    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 03:09:29.939575 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.939557    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 03:09:29.939668 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.939594    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 03:09:29.939668 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.939609    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientPID"
Aug 04 03:09:29.942131 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.942112    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 03:09:29.942232 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.942222    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal"
Aug 04 03:09:29.942303 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.942251    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 03:09:29.942976 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.942941    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 03:09:29.942976 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.942964    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 03:09:29.943106 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.942992    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientPID"
Aug 04 03:09:29.943106 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.943030    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 03:09:29.943106 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.943061    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 03:09:29.943106 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.943076    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientPID"
Aug 04 03:09:29.945429 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.945413    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal"
Aug 04 03:09:29.945499 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.945447    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 03:09:29.946191 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.946174    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 03:09:29.946266 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.946202    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 03:09:29.946266 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.946217    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientPID"
Aug 04 03:09:29.969881 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:29.969852    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-99.ec2.internal\" not found" node="ip-10-0-138-99.ec2.internal"
Aug 04 03:09:29.974479 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:29.974460    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-99.ec2.internal\" not found" node="ip-10-0-138-99.ec2.internal"
Aug 04 03:09:29.983921 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.983898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3e36ae9120ed68114f2482f70b5b344-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal\" (UID: \"c3e36ae9120ed68114f2482f70b5b344\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal"
Aug 04 03:09:29.984016 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.983931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3e36ae9120ed68114f2482f70b5b344-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal\" (UID: \"c3e36ae9120ed68114f2482f70b5b344\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal"
Aug 04 03:09:29.984016 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:29.983960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3fb028bcdaa2f5ddcb427d9e021fdc0b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-99.ec2.internal\" (UID: \"3fb028bcdaa2f5ddcb427d9e021fdc0b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal"
Aug 04 03:09:29.987198 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:29.987180    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Aug 04 03:09:30.085004 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:30.084951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3e36ae9120ed68114f2482f70b5b344-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal\" (UID: \"c3e36ae9120ed68114f2482f70b5b344\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal"
Aug 04 03:09:30.085004 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:30.085006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3e36ae9120ed68114f2482f70b5b344-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal\" (UID: \"c3e36ae9120ed68114f2482f70b5b344\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal"
Aug 04 03:09:30.085269 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:30.085059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3fb028bcdaa2f5ddcb427d9e021fdc0b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-99.ec2.internal\" (UID: \"3fb028bcdaa2f5ddcb427d9e021fdc0b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal"
Aug 04 03:09:30.085269 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:30.085067    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3e36ae9120ed68114f2482f70b5b344-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal\" (UID: \"c3e36ae9120ed68114f2482f70b5b344\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal"
Aug 04 03:09:30.085269 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:30.085095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3e36ae9120ed68114f2482f70b5b344-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal\" (UID: \"c3e36ae9120ed68114f2482f70b5b344\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal"
Aug 04 03:09:30.085269 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:30.085135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3fb028bcdaa2f5ddcb427d9e021fdc0b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-99.ec2.internal\" (UID: \"3fb028bcdaa2f5ddcb427d9e021fdc0b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal"
Aug 04 03:09:30.088301 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:30.088281    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Aug 04 03:09:30.188607 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:30.188514    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Aug 04 03:09:30.278660 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:30.278612    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal"
Aug 04 03:09:30.278806 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:30.278612    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal"
Aug 04 03:09:30.289018 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:30.288996    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Aug 04 03:09:30.390039 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:30.389998    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Aug 04 03:09:30.491128 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:30.491061    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Aug 04 03:09:30.591264 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:30.591237    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Aug 04 03:09:30.625437 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:30.625408    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Aug 04 03:09:30.625903 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:30.625555    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 04 03:09:30.655493 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:30.655459    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 03:09:30.692107 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:30.692075    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Aug 04 03:09:30.714966 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:30.714924    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 03:04:29 +0000 UTC" deadline="2028-04-06 18:49:21.98649567 +0000 UTC"
Aug 04 03:09:30.715039 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:30.714961    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14679h39m51.27153845s"
Aug 04 03:09:30.792888 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:30.792811    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Aug 04 03:09:30.893547 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:30.893512    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Aug 04 03:09:30.949677 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:30.949644    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb028bcdaa2f5ddcb427d9e021fdc0b.slice/crio-92795dfd48e71d9b04c26c9bacf86adea3b03de78cf6f7977c6adcead7c9ba15 WatchSource:0}: Error finding container 92795dfd48e71d9b04c26c9bacf86adea3b03de78cf6f7977c6adcead7c9ba15: Status 404 returned error can't find the container with id 92795dfd48e71d9b04c26c9bacf86adea3b03de78cf6f7977c6adcead7c9ba15
Aug 04 03:09:30.956696 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:30.956678    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 03:09:30.962501 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:30.962478    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e36ae9120ed68114f2482f70b5b344.slice/crio-3dc6d7b5e340b070ebe626a61b6ddfac8e1b4ad34bd5bd441462f4e480692db6 WatchSource:0}: Error finding container 3dc6d7b5e340b070ebe626a61b6ddfac8e1b4ad34bd5bd441462f4e480692db6: Status 404 returned error can't find the container with id 3dc6d7b5e340b070ebe626a61b6ddfac8e1b4ad34bd5bd441462f4e480692db6
Aug 04 03:09:30.994710 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:30.994675    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Aug 04 03:09:30.995137 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:30.995120    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 03:09:31.077561 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.077485    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 03:09:31.081812 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.081796    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal"
Aug 04 03:09:31.097995 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.097969    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 03:09:31.100020 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.100005    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal"
Aug 04 03:09:31.112636 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.112614    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 03:09:31.663813 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.663780    2579 apiserver.go:52] "Watching apiserver"
Aug 04 03:09:31.676265 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.676205    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Aug 04 03:09:31.677787 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.677755    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d","openshift-dns/node-resolver-57qsq","openshift-image-registry/node-ca-gnp8m","openshift-multus/multus-additional-cni-plugins-nbbch","openshift-multus/multus-h5jxp","openshift-multus/network-metrics-daemon-c5d74","kube-system/global-pull-secret-syncer-gcg5l","openshift-cluster-node-tuning-operator/tuned-zr5zh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal","openshift-network-diagnostics/network-check-target-qnvmg","openshift-network-operator/iptables-alerter-tqvgl","openshift-ovn-kubernetes/ovnkube-node-4zfwh","kube-system/konnectivity-agent-v6hhn"]
Aug 04 03:09:31.681575 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.681546    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:31.681684 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:31.681646    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:09:31.682066 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.681847    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Aug 04 03:09:31.685511 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.685199    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.687545 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.687524    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-57qsq"
Aug 04 03:09:31.688479 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.688455    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Aug 04 03:09:31.688738 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.688717    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rrlb7\""
Aug 04 03:09:31.688977 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.688957    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Aug 04 03:09:31.689364 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.689344    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Aug 04 03:09:31.689908 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.689613    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Aug 04 03:09:31.689908 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.689757    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Aug 04 03:09:31.690096 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.690025    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5dskl\""
Aug 04 03:09:31.690813 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.690328    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Aug 04 03:09:31.690813 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.690633    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Aug 04 03:09:31.690813 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.690639    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Aug 04 03:09:31.691995 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.690893    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gnp8m"
Aug 04 03:09:31.694284 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.693075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.694284 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.693359    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Aug 04 03:09:31.694284 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.693980    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Aug 04 03:09:31.694284 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.694057    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lcmcw\""
Aug 04 03:09:31.694284 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.694066    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Aug 04 03:09:31.695527 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.695502    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 03:09:31.696298 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.695743    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Aug 04 03:09:31.696298 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.696215    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Aug 04 03:09:31.696505 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.696483    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-brxhp\""
Aug 04 03:09:31.697152 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.696682    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Aug 04 03:09:31.697152 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.696882    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Aug 04 03:09:31.697152 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697086    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Aug 04 03:09:31.697381 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697317    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-systemd-units\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.697381 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697352    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-run-ovn-kubernetes\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.697495 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.697495 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7df4e9d6-1779-4898-a32b-3b6e7d569896-ovn-node-metrics-cert\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.697495 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-run-ovn\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.697495 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697460    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs\") pod \"network-metrics-daemon-c5d74\" (UID: \"b39560be-b9af-4ff7-8a5b-7882c66a03d8\") " pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:31.697495 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697482    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-slash\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.697747 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-var-lib-openvswitch\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.697747 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-log-socket\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.697747 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697549    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-cni-bin\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.697747 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7df4e9d6-1779-4898-a32b-3b6e7d569896-ovnkube-config\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.697747 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7df4e9d6-1779-4898-a32b-3b6e7d569896-env-overrides\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.697747 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-run-systemd\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.697747 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697624    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-run-openvswitch\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.697747 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-cni-netd\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.697747 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m4cj\" (UniqueName: \"kubernetes.io/projected/7df4e9d6-1779-4898-a32b-3b6e7d569896-kube-api-access-2m4cj\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.697747 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w26dh\" (UniqueName: \"kubernetes.io/projected/b39560be-b9af-4ff7-8a5b-7882c66a03d8-kube-api-access-w26dh\") pod \"network-metrics-daemon-c5d74\" (UID: \"b39560be-b9af-4ff7-8a5b-7882c66a03d8\") " pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:31.698399 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-run-netns\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.698399 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697790    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-etc-openvswitch\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.698399 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-node-log\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.698399 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697851    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7df4e9d6-1779-4898-a32b-3b6e7d569896-ovnkube-script-lib\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.698399 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.697878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-kubelet\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.698677 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.698663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.700038 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.699468    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.701772 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.701308    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Aug 04 03:09:31.701772 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.701590    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qpfc8\""
Aug 04 03:09:31.702865 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.702119    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Aug 04 03:09:31.702865 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.702228    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bqgvj\""
Aug 04 03:09:31.702865 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.702474    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v6hhn"
Aug 04 03:09:31.703595 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.703308    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Aug 04 03:09:31.706885 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.705103    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Aug 04 03:09:31.706885 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.705416    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Aug 04 03:09:31.706885 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.705454    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-87ldg\""
Aug 04 03:09:31.708717 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.708068    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.711680 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.710365    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:31.711680 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:31.710444    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcg5l" podUID="54b8b42f-6de1-4f5a-8115-50f8b2ff573f"
Aug 04 03:09:31.711680 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.711135    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k4jxk\""
Aug 04 03:09:31.711680 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.711261    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Aug 04 03:09:31.711680 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.711381    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Aug 04 03:09:31.711680 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.711487    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Aug 04 03:09:31.714578 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.713381    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:31.714578 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:31.713448    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnvmg" podUID="b2d2669b-897a-4fdf-8df5-0cd56e7d47a5"
Aug 04 03:09:31.721209 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.717445    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tqvgl"
Aug 04 03:09:31.723782 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.722753    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zzx7d\""
Aug 04 03:09:31.723782 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.723191    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Aug 04 03:09:31.723782 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.723305    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nz6tt"
Aug 04 03:09:31.723782 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.723464    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Aug 04 03:09:31.723782 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.723564    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Aug 04 03:09:31.729343 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.728897    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nz6tt"
Aug 04 03:09:31.783424 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.783391    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Aug 04 03:09:31.798709 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.798660    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-run-netns\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.798709 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.798705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-node-log\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.798946 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.798739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59a54985-bd71-4de1-83d7-ef1f131cff79-sys-fs\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.798946 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.798766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-modprobe-d\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.798946 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.798785    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-run-netns\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.798946 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.798793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-host\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.798946 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.798842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-node-log\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.798946 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.798861    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb1dbb4a-c31f-4485-b847-9a54379f39d1-konnectivity-ca\") pod \"konnectivity-agent-v6hhn\" (UID: \"cb1dbb4a-c31f-4485-b847-9a54379f39d1\") " pod="kube-system/konnectivity-agent-v6hhn"
Aug 04 03:09:31.798946 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.798922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-kubelet\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.798946 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.798950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-systemd-units\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.799389 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-kubelet\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.799389 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c429971-1d7d-4d34-b0fb-c2081a928855-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.799389 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799063    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-systemd-units\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.799389 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-kubernetes\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.799389 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-lib-modules\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.799389 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799137    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-tuned\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.799389 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799162    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret\") pod \"global-pull-secret-syncer-gcg5l\" (UID: \"54b8b42f-6de1-4f5a-8115-50f8b2ff573f\") " pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:31.799389 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799195    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9-host-slash\") pod \"iptables-alerter-tqvgl\" (UID: \"dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9\") " pod="openshift-network-operator/iptables-alerter-tqvgl"
Aug 04 03:09:31.799389 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-var-lib-openvswitch\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.799389 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7df4e9d6-1779-4898-a32b-3b6e7d569896-ovnkube-config\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.799389 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-sysctl-conf\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.799389 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-run\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.799389 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799350    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-var-lib-openvswitch\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.799389 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799391    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-dbus\") pod \"global-pull-secret-syncer-gcg5l\" (UID: \"54b8b42f-6de1-4f5a-8115-50f8b2ff573f\") " pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:31.800082 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tstqk\" (UniqueName: \"kubernetes.io/projected/dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9-kube-api-access-tstqk\") pod \"iptables-alerter-tqvgl\" (UID: \"dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9\") " pod="openshift-network-operator/iptables-alerter-tqvgl"
Aug 04 03:09:31.800082 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8-tmp-dir\") pod \"node-resolver-57qsq\" (UID: \"d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8\") " pod="openshift-dns/node-resolver-57qsq"
Aug 04 03:09:31.800082 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799485    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c429971-1d7d-4d34-b0fb-c2081a928855-cni-binary-copy\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.800082 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2c429971-1d7d-4d34-b0fb-c2081a928855-os-release\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.800082 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-sys\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.800082 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vvqt\" (UniqueName: \"kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt\") pod \"network-check-target-qnvmg\" (UID: \"b2d2669b-897a-4fdf-8df5-0cd56e7d47a5\") " pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:31.800082 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-run-ovn-kubernetes\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.800082 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj7vn\" (UniqueName: \"kubernetes.io/projected/d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8-kube-api-access-sj7vn\") pod \"node-resolver-57qsq\" (UID: \"d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8\") " pod="openshift-dns/node-resolver-57qsq"
Aug 04 03:09:31.800082 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-run-ovn-kubernetes\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.800082 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799722    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-multus-socket-dir-parent\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.800082 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-hostroot\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.800082 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8g7c\" (UniqueName: \"kubernetes.io/projected/2c429971-1d7d-4d34-b0fb-c2081a928855-kube-api-access-p8g7c\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.800082 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799810    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs\") pod \"network-metrics-daemon-c5d74\" (UID: \"b39560be-b9af-4ff7-8a5b-7882c66a03d8\") " pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:31.800082 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-slash\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.800082 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799879    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f76ca57-4490-4583-85a2-95198fb43ffd-host\") pod \"node-ca-gnp8m\" (UID: \"5f76ca57-4490-4583-85a2-95198fb43ffd\") " pod="openshift-image-registry/node-ca-gnp8m"
Aug 04 03:09:31.800082 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-host-var-lib-cni-bin\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.800082 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799911    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-slash\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.800975 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799932    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-var-lib-kubelet\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.800975 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:31.799921    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 03:09:31.800975 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.799959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-run-systemd\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.800975 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-run-openvswitch\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.800975 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:31.800028    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs podName:b39560be-b9af-4ff7-8a5b-7882c66a03d8 nodeName:}" failed. No retries permitted until 2026-08-04 03:09:32.300002533 +0000 UTC m=+2.979789650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs") pod "network-metrics-daemon-c5d74" (UID: "b39560be-b9af-4ff7-8a5b-7882c66a03d8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 03:09:31.800975 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-run-systemd\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.800975 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-run-openvswitch\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.800975 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4cj\" (UniqueName: \"kubernetes.io/projected/7df4e9d6-1779-4898-a32b-3b6e7d569896-kube-api-access-2m4cj\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.800975 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800081    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7df4e9d6-1779-4898-a32b-3b6e7d569896-ovnkube-config\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.800975 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800097    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-host-run-multus-certs\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.800975 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w26dh\" (UniqueName: \"kubernetes.io/projected/b39560be-b9af-4ff7-8a5b-7882c66a03d8-kube-api-access-w26dh\") pod \"network-metrics-daemon-c5d74\" (UID: \"b39560be-b9af-4ff7-8a5b-7882c66a03d8\") " pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:31.800975 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-etc-openvswitch\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.800975 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7df4e9d6-1779-4898-a32b-3b6e7d569896-ovnkube-script-lib\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.800975 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8-hosts-file\") pod \"node-resolver-57qsq\" (UID: \"d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8\") " pod="openshift-dns/node-resolver-57qsq"
Aug 04 03:09:31.800975 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-host-run-netns\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.800975 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-etc-openvswitch\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.800975 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800284    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-host-var-lib-kubelet\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.801901 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c429971-1d7d-4d34-b0fb-c2081a928855-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.801901 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.801901 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htfc2\" (UniqueName: \"kubernetes.io/projected/fa970b56-8654-40d5-b836-005f3c50b89f-kube-api-access-htfc2\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.801901 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-systemd\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.801901 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800474    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.801901 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-tmp\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.801901 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-os-release\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.801901 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa970b56-8654-40d5-b836-005f3c50b89f-cni-binary-copy\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.801901 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59a54985-bd71-4de1-83d7-ef1f131cff79-socket-dir\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.801901 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59a54985-bd71-4de1-83d7-ef1f131cff79-registration-dir\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.801901 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-run-ovn\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.801901 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-log-socket\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.801901 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7df4e9d6-1779-4898-a32b-3b6e7d569896-env-overrides\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.801901 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f76ca57-4490-4583-85a2-95198fb43ffd-serviceca\") pod \"node-ca-gnp8m\" (UID: \"5f76ca57-4490-4583-85a2-95198fb43ffd\") " pod="openshift-image-registry/node-ca-gnp8m"
Aug 04 03:09:31.801901 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800733    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-run-ovn\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.801901 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800740    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-system-cni-dir\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.801901 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800747    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7df4e9d6-1779-4898-a32b-3b6e7d569896-ovnkube-script-lib\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.802816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800852    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-multus-conf-dir\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.802816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800863    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-log-socket\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.802816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59a54985-bd71-4de1-83d7-ef1f131cff79-device-dir\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.802816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59a54985-bd71-4de1-83d7-ef1f131cff79-etc-selinux\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.802816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c429971-1d7d-4d34-b0fb-c2081a928855-system-cni-dir\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.802816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.800966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-multus-cni-dir\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.802816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-cnibin\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.802816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801023    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c429971-1d7d-4d34-b0fb-c2081a928855-cnibin\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.802816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc2p6\" (UniqueName: \"kubernetes.io/projected/5f76ca57-4490-4583-85a2-95198fb43ffd-kube-api-access-mc2p6\") pod \"node-ca-gnp8m\" (UID: \"5f76ca57-4490-4583-85a2-95198fb43ffd\") " pod="openshift-image-registry/node-ca-gnp8m"
Aug 04 03:09:31.802816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-sysconfig\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.802816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801097    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9-iptables-alerter-script\") pod \"iptables-alerter-tqvgl\" (UID: \"dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9\") " pod="openshift-network-operator/iptables-alerter-tqvgl"
Aug 04 03:09:31.802816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7df4e9d6-1779-4898-a32b-3b6e7d569896-ovn-node-metrics-cert\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.802816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59a54985-bd71-4de1-83d7-ef1f131cff79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.802816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnzrv\" (UniqueName: \"kubernetes.io/projected/59a54985-bd71-4de1-83d7-ef1f131cff79-kube-api-access-lnzrv\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.802816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7df4e9d6-1779-4898-a32b-3b6e7d569896-env-overrides\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.802816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdgpw\" (UniqueName: \"kubernetes.io/projected/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-kube-api-access-gdgpw\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.803656 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-cni-bin\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.803656 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-host-var-lib-cni-multus\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.803656 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-cni-bin\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.803656 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fa970b56-8654-40d5-b836-005f3c50b89f-multus-daemon-config\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.803656 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-etc-kubernetes\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.803656 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801368    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-kubelet-config\") pod \"global-pull-secret-syncer-gcg5l\" (UID: \"54b8b42f-6de1-4f5a-8115-50f8b2ff573f\") " pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:31.803656 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-cni-netd\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.803656 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801443    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-host-run-k8s-cni-cncf-io\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.803656 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c429971-1d7d-4d34-b0fb-c2081a928855-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.803656 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801519    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-sysctl-d\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.803656 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb1dbb4a-c31f-4485-b847-9a54379f39d1-agent-certs\") pod \"konnectivity-agent-v6hhn\" (UID: \"cb1dbb4a-c31f-4485-b847-9a54379f39d1\") " pod="kube-system/konnectivity-agent-v6hhn"
Aug 04 03:09:31.803656 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801541    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Aug 04 03:09:31.803656 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.801580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7df4e9d6-1779-4898-a32b-3b6e7d569896-host-cni-netd\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.806592 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.806089    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7df4e9d6-1779-4898-a32b-3b6e7d569896-ovn-node-metrics-cert\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.809298 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.809179    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m4cj\" (UniqueName: \"kubernetes.io/projected/7df4e9d6-1779-4898-a32b-3b6e7d569896-kube-api-access-2m4cj\") pod \"ovnkube-node-4zfwh\" (UID: \"7df4e9d6-1779-4898-a32b-3b6e7d569896\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:31.814731 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.814707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w26dh\" (UniqueName: \"kubernetes.io/projected/b39560be-b9af-4ff7-8a5b-7882c66a03d8-kube-api-access-w26dh\") pod \"network-metrics-daemon-c5d74\" (UID: \"b39560be-b9af-4ff7-8a5b-7882c66a03d8\") " pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:31.845310 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.845252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal" event={"ID":"c3e36ae9120ed68114f2482f70b5b344","Type":"ContainerStarted","Data":"3dc6d7b5e340b070ebe626a61b6ddfac8e1b4ad34bd5bd441462f4e480692db6"}
Aug 04 03:09:31.846978 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.846935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal" event={"ID":"3fb028bcdaa2f5ddcb427d9e021fdc0b","Type":"ContainerStarted","Data":"92795dfd48e71d9b04c26c9bacf86adea3b03de78cf6f7977c6adcead7c9ba15"}
Aug 04 03:09:31.902036 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.901941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-dbus\") pod \"global-pull-secret-syncer-gcg5l\" (UID: \"54b8b42f-6de1-4f5a-8115-50f8b2ff573f\") " pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:31.902036 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tstqk\" (UniqueName: \"kubernetes.io/projected/dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9-kube-api-access-tstqk\") pod \"iptables-alerter-tqvgl\" (UID: \"dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9\") " pod="openshift-network-operator/iptables-alerter-tqvgl"
Aug 04 03:09:31.902302 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8-tmp-dir\") pod \"node-resolver-57qsq\" (UID: \"d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8\") " pod="openshift-dns/node-resolver-57qsq"
Aug 04 03:09:31.902302 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c429971-1d7d-4d34-b0fb-c2081a928855-cni-binary-copy\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.905230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2c429971-1d7d-4d34-b0fb-c2081a928855-os-release\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.905230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-sys\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.905230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vvqt\" (UniqueName: \"kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt\") pod \"network-check-target-qnvmg\" (UID: \"b2d2669b-897a-4fdf-8df5-0cd56e7d47a5\") " pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:31.905230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902562    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-dbus\") pod \"global-pull-secret-syncer-gcg5l\" (UID: \"54b8b42f-6de1-4f5a-8115-50f8b2ff573f\") " pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:31.905230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj7vn\" (UniqueName: \"kubernetes.io/projected/d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8-kube-api-access-sj7vn\") pod \"node-resolver-57qsq\" (UID: \"d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8\") " pod="openshift-dns/node-resolver-57qsq"
Aug 04 03:09:31.905230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-multus-socket-dir-parent\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.905230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902632    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-hostroot\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.905230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902635    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-sys\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.905230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902643    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8-tmp-dir\") pod \"node-resolver-57qsq\" (UID: \"d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8\") " pod="openshift-dns/node-resolver-57qsq"
Aug 04 03:09:31.905230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8g7c\" (UniqueName: \"kubernetes.io/projected/2c429971-1d7d-4d34-b0fb-c2081a928855-kube-api-access-p8g7c\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.905230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f76ca57-4490-4583-85a2-95198fb43ffd-host\") pod \"node-ca-gnp8m\" (UID: \"5f76ca57-4490-4583-85a2-95198fb43ffd\") " pod="openshift-image-registry/node-ca-gnp8m"
Aug 04 03:09:31.905230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c429971-1d7d-4d34-b0fb-c2081a928855-cni-binary-copy\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.905230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-host-var-lib-cni-bin\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.905230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902703    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2c429971-1d7d-4d34-b0fb-c2081a928855-os-release\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.905230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-var-lib-kubelet\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.905230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-host-run-multus-certs\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.905230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8-hosts-file\") pod \"node-resolver-57qsq\" (UID: \"d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8\") " pod="openshift-dns/node-resolver-57qsq"
Aug 04 03:09:31.905230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-multus-socket-dir-parent\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.906350 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-host-run-netns\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.906350 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-host-var-lib-kubelet\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.906350 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-host-var-lib-cni-bin\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.906350 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902932    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-hostroot\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.906350 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902938    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-host-var-lib-kubelet\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.906350 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c429971-1d7d-4d34-b0fb-c2081a928855-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.906350 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f76ca57-4490-4583-85a2-95198fb43ffd-host\") pod \"node-ca-gnp8m\" (UID: \"5f76ca57-4490-4583-85a2-95198fb43ffd\") " pod="openshift-image-registry/node-ca-gnp8m"
Aug 04 03:09:31.906350 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903017    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-host-run-multus-certs\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.906350 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htfc2\" (UniqueName: \"kubernetes.io/projected/fa970b56-8654-40d5-b836-005f3c50b89f-kube-api-access-htfc2\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.906350 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.902802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-var-lib-kubelet\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.906350 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-systemd\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.906350 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903126    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8-hosts-file\") pod \"node-resolver-57qsq\" (UID: \"d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8\") " pod="openshift-dns/node-resolver-57qsq"
Aug 04 03:09:31.906350 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-tmp\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.906350 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903185    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-os-release\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.906350 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa970b56-8654-40d5-b836-005f3c50b89f-cni-binary-copy\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.906350 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59a54985-bd71-4de1-83d7-ef1f131cff79-socket-dir\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.906350 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59a54985-bd71-4de1-83d7-ef1f131cff79-registration-dir\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.906350 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f76ca57-4490-4583-85a2-95198fb43ffd-serviceca\") pod \"node-ca-gnp8m\" (UID: \"5f76ca57-4490-4583-85a2-95198fb43ffd\") " pod="openshift-image-registry/node-ca-gnp8m"
Aug 04 03:09:31.907274 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-system-cni-dir\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.907274 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-multus-conf-dir\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.907274 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59a54985-bd71-4de1-83d7-ef1f131cff79-device-dir\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.907274 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903417    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59a54985-bd71-4de1-83d7-ef1f131cff79-etc-selinux\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.907274 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c429971-1d7d-4d34-b0fb-c2081a928855-system-cni-dir\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.907274 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-multus-cni-dir\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.907274 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-cnibin\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.907274 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c429971-1d7d-4d34-b0fb-c2081a928855-cnibin\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.907274 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903551    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc2p6\" (UniqueName: \"kubernetes.io/projected/5f76ca57-4490-4583-85a2-95198fb43ffd-kube-api-access-mc2p6\") pod \"node-ca-gnp8m\" (UID: \"5f76ca57-4490-4583-85a2-95198fb43ffd\") " pod="openshift-image-registry/node-ca-gnp8m"
Aug 04 03:09:31.907274 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-sysconfig\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.907274 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9-iptables-alerter-script\") pod \"iptables-alerter-tqvgl\" (UID: \"dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9\") " pod="openshift-network-operator/iptables-alerter-tqvgl"
Aug 04 03:09:31.907274 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59a54985-bd71-4de1-83d7-ef1f131cff79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.907274 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnzrv\" (UniqueName: \"kubernetes.io/projected/59a54985-bd71-4de1-83d7-ef1f131cff79-kube-api-access-lnzrv\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.907274 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdgpw\" (UniqueName: \"kubernetes.io/projected/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-kube-api-access-gdgpw\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.907274 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-host-var-lib-cni-multus\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.907274 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fa970b56-8654-40d5-b836-005f3c50b89f-multus-daemon-config\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.907274 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903775    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c429971-1d7d-4d34-b0fb-c2081a928855-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.908201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-etc-kubernetes\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.908201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-kubelet-config\") pod \"global-pull-secret-syncer-gcg5l\" (UID: \"54b8b42f-6de1-4f5a-8115-50f8b2ff573f\") " pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:31.908201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-etc-kubernetes\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.908201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903854    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-host-run-k8s-cni-cncf-io\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.908201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-cnibin\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.908201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903912    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-kubelet-config\") pod \"global-pull-secret-syncer-gcg5l\" (UID: \"54b8b42f-6de1-4f5a-8115-50f8b2ff573f\") " pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:31.908201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903925    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-multus-cni-dir\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.908201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c429971-1d7d-4d34-b0fb-c2081a928855-cnibin\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.908201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903961    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-host-run-k8s-cni-cncf-io\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.908201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.903988    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-host-run-netns\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.908201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.904019    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-os-release\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.908201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.904235    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-sysconfig\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.908201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.904344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-host-var-lib-cni-multus\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.908201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.904557    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa970b56-8654-40d5-b836-005f3c50b89f-cni-binary-copy\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.908201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.904681    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59a54985-bd71-4de1-83d7-ef1f131cff79-socket-dir\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.908201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.904740    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59a54985-bd71-4de1-83d7-ef1f131cff79-registration-dir\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.908201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.904778    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9-iptables-alerter-script\") pod \"iptables-alerter-tqvgl\" (UID: \"dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9\") " pod="openshift-network-operator/iptables-alerter-tqvgl"
Aug 04 03:09:31.908201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.904800    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fa970b56-8654-40d5-b836-005f3c50b89f-multus-daemon-config\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.909085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.904853    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59a54985-bd71-4de1-83d7-ef1f131cff79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.909085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.904907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59a54985-bd71-4de1-83d7-ef1f131cff79-device-dir\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.909085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.904957    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-system-cni-dir\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.909085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905007    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa970b56-8654-40d5-b836-005f3c50b89f-multus-conf-dir\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.909085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-systemd\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.909085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c429971-1d7d-4d34-b0fb-c2081a928855-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.909085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c429971-1d7d-4d34-b0fb-c2081a928855-system-cni-dir\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.909085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-sysctl-d\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.909085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905084    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f76ca57-4490-4583-85a2-95198fb43ffd-serviceca\") pod \"node-ca-gnp8m\" (UID: \"5f76ca57-4490-4583-85a2-95198fb43ffd\") " pod="openshift-image-registry/node-ca-gnp8m"
Aug 04 03:09:31.909085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb1dbb4a-c31f-4485-b847-9a54379f39d1-agent-certs\") pod \"konnectivity-agent-v6hhn\" (UID: \"cb1dbb4a-c31f-4485-b847-9a54379f39d1\") " pod="kube-system/konnectivity-agent-v6hhn"
Aug 04 03:09:31.909085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59a54985-bd71-4de1-83d7-ef1f131cff79-sys-fs\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.909085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59a54985-bd71-4de1-83d7-ef1f131cff79-etc-selinux\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.909085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-modprobe-d\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.909085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905214    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59a54985-bd71-4de1-83d7-ef1f131cff79-sys-fs\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:31.909085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-host\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.909085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905341    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb1dbb4a-c31f-4485-b847-9a54379f39d1-konnectivity-ca\") pod \"konnectivity-agent-v6hhn\" (UID: \"cb1dbb4a-c31f-4485-b847-9a54379f39d1\") " pod="kube-system/konnectivity-agent-v6hhn"
Aug 04 03:09:31.909085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905397    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-host\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.909932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905454    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c429971-1d7d-4d34-b0fb-c2081a928855-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.909932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905471    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-sysctl-d\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.909932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905483    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-kubernetes\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.909932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905510    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-lib-modules\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.909932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-modprobe-d\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.909932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-tuned\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.909932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret\") pod \"global-pull-secret-syncer-gcg5l\" (UID: \"54b8b42f-6de1-4f5a-8115-50f8b2ff573f\") " pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:31.909932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905581    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c429971-1d7d-4d34-b0fb-c2081a928855-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.909932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9-host-slash\") pod \"iptables-alerter-tqvgl\" (UID: \"dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9\") " pod="openshift-network-operator/iptables-alerter-tqvgl"
Aug 04 03:09:31.909932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-sysctl-conf\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.909932 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:31.905665    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 03:09:31.909932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-run\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.909932 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:31.905733    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret podName:54b8b42f-6de1-4f5a-8115-50f8b2ff573f nodeName:}" failed. No retries permitted until 2026-08-04 03:09:32.405714146 +0000 UTC m=+3.085501258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret") pod "global-pull-secret-syncer-gcg5l" (UID: "54b8b42f-6de1-4f5a-8115-50f8b2ff573f") : object "kube-system"/"original-pull-secret" not registered
Aug 04 03:09:31.909932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905755    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-run\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.909932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905792    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9-host-slash\") pod \"iptables-alerter-tqvgl\" (UID: \"dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9\") " pod="openshift-network-operator/iptables-alerter-tqvgl"
Aug 04 03:09:31.909932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905915    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-sysctl-conf\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.909932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.905943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-kubernetes\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.910818 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.906021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb1dbb4a-c31f-4485-b847-9a54379f39d1-konnectivity-ca\") pod \"konnectivity-agent-v6hhn\" (UID: \"cb1dbb4a-c31f-4485-b847-9a54379f39d1\") " pod="kube-system/konnectivity-agent-v6hhn"
Aug 04 03:09:31.910818 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.906068    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-lib-modules\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.910818 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.906151    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c429971-1d7d-4d34-b0fb-c2081a928855-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.910818 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.908315    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb1dbb4a-c31f-4485-b847-9a54379f39d1-agent-certs\") pod \"konnectivity-agent-v6hhn\" (UID: \"cb1dbb4a-c31f-4485-b847-9a54379f39d1\") " pod="kube-system/konnectivity-agent-v6hhn"
Aug 04 03:09:31.910818 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.908526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-etc-tuned\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.910818 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.909538    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-tmp\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.912533 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:31.912298    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 03:09:31.912533 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:31.912321    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 03:09:31.912533 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:31.912335    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4vvqt for pod openshift-network-diagnostics/network-check-target-qnvmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 03:09:31.912533 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:31.912406    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt podName:b2d2669b-897a-4fdf-8df5-0cd56e7d47a5 nodeName:}" failed. No retries permitted until 2026-08-04 03:09:32.412388744 +0000 UTC m=+3.092175858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4vvqt" (UniqueName: "kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt") pod "network-check-target-qnvmg" (UID: "b2d2669b-897a-4fdf-8df5-0cd56e7d47a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 03:09:31.914472 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.914414    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc2p6\" (UniqueName: \"kubernetes.io/projected/5f76ca57-4490-4583-85a2-95198fb43ffd-kube-api-access-mc2p6\") pod \"node-ca-gnp8m\" (UID: \"5f76ca57-4490-4583-85a2-95198fb43ffd\") " pod="openshift-image-registry/node-ca-gnp8m"
Aug 04 03:09:31.915184 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.915137    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdgpw\" (UniqueName: \"kubernetes.io/projected/ff980ac9-f6a3-42f9-b1e9-36419f3d927a-kube-api-access-gdgpw\") pod \"tuned-zr5zh\" (UID: \"ff980ac9-f6a3-42f9-b1e9-36419f3d927a\") " pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:31.916121 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.916045    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj7vn\" (UniqueName: \"kubernetes.io/projected/d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8-kube-api-access-sj7vn\") pod \"node-resolver-57qsq\" (UID: \"d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8\") " pod="openshift-dns/node-resolver-57qsq"
Aug 04 03:09:31.916569 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.916307    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tstqk\" (UniqueName: \"kubernetes.io/projected/dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9-kube-api-access-tstqk\") pod \"iptables-alerter-tqvgl\" (UID: \"dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9\") " pod="openshift-network-operator/iptables-alerter-tqvgl"
Aug 04 03:09:31.916732 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.916707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htfc2\" (UniqueName: \"kubernetes.io/projected/fa970b56-8654-40d5-b836-005f3c50b89f-kube-api-access-htfc2\") pod \"multus-h5jxp\" (UID: \"fa970b56-8654-40d5-b836-005f3c50b89f\") " pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:31.917749 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.917713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8g7c\" (UniqueName: \"kubernetes.io/projected/2c429971-1d7d-4d34-b0fb-c2081a928855-kube-api-access-p8g7c\") pod \"multus-additional-cni-plugins-nbbch\" (UID: \"2c429971-1d7d-4d34-b0fb-c2081a928855\") " pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:31.918268 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:31.918214    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnzrv\" (UniqueName: \"kubernetes.io/projected/59a54985-bd71-4de1-83d7-ef1f131cff79-kube-api-access-lnzrv\") pod \"aws-ebs-csi-driver-node-vd42d\" (UID: \"59a54985-bd71-4de1-83d7-ef1f131cff79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:32.001117 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.001079    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:32.013016 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.012900    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-57qsq"
Aug 04 03:09:32.015075 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:32.015046    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df4e9d6_1779_4898_a32b_3b6e7d569896.slice/crio-0fc7e2b51fad0799f1c9d9cb360a8809f2c5093501f1d41d813850f22169314f WatchSource:0}: Error finding container 0fc7e2b51fad0799f1c9d9cb360a8809f2c5093501f1d41d813850f22169314f: Status 404 returned error can't find the container with id 0fc7e2b51fad0799f1c9d9cb360a8809f2c5093501f1d41d813850f22169314f
Aug 04 03:09:32.023229 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:32.023201    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8bb4fcc_2c06_4bb0_b23b_249bda7fa7b8.slice/crio-e53bef1b21d916d8c68e82bce43a2ed48ac0bc6de3c5ce9cbcccf4b205a88490 WatchSource:0}: Error finding container e53bef1b21d916d8c68e82bce43a2ed48ac0bc6de3c5ce9cbcccf4b205a88490: Status 404 returned error can't find the container with id e53bef1b21d916d8c68e82bce43a2ed48ac0bc6de3c5ce9cbcccf4b205a88490
Aug 04 03:09:32.028234 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.028208    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gnp8m"
Aug 04 03:09:32.036703 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.036678    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nbbch"
Aug 04 03:09:32.040079 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:32.039894    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f76ca57_4490_4583_85a2_95198fb43ffd.slice/crio-afeb41c6a0129c7b56c0676f5fd51805f467e3032ddcc0d9be5890d783619f19 WatchSource:0}: Error finding container afeb41c6a0129c7b56c0676f5fd51805f467e3032ddcc0d9be5890d783619f19: Status 404 returned error can't find the container with id afeb41c6a0129c7b56c0676f5fd51805f467e3032ddcc0d9be5890d783619f19
Aug 04 03:09:32.043905 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.043758    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h5jxp"
Aug 04 03:09:32.047937 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:32.047908    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c429971_1d7d_4d34_b0fb_c2081a928855.slice/crio-c498609a7de8f8851e644b6d9c3f5ebedbd55a04442f7897ac288c91db8b1450 WatchSource:0}: Error finding container c498609a7de8f8851e644b6d9c3f5ebedbd55a04442f7897ac288c91db8b1450: Status 404 returned error can't find the container with id c498609a7de8f8851e644b6d9c3f5ebedbd55a04442f7897ac288c91db8b1450
Aug 04 03:09:32.052765 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.052739    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zr5zh"
Aug 04 03:09:32.057368 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:32.057319    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa970b56_8654_40d5_b836_005f3c50b89f.slice/crio-ce22dff1d85baed9a70d5f06b4722804ae344a303e090a3da8a211595e61d700 WatchSource:0}: Error finding container ce22dff1d85baed9a70d5f06b4722804ae344a303e090a3da8a211595e61d700: Status 404 returned error can't find the container with id ce22dff1d85baed9a70d5f06b4722804ae344a303e090a3da8a211595e61d700
Aug 04 03:09:32.060503 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.060469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v6hhn"
Aug 04 03:09:32.066028 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:32.065999    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff980ac9_f6a3_42f9_b1e9_36419f3d927a.slice/crio-03449524e82c8620a4193b8f61c290d9c9ca78a4229ffffec8c65f414f6d8116 WatchSource:0}: Error finding container 03449524e82c8620a4193b8f61c290d9c9ca78a4229ffffec8c65f414f6d8116: Status 404 returned error can't find the container with id 03449524e82c8620a4193b8f61c290d9c9ca78a4229ffffec8c65f414f6d8116
Aug 04 03:09:32.069024 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.069005    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d"
Aug 04 03:09:32.073043 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:32.073016    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1dbb4a_c31f_4485_b847_9a54379f39d1.slice/crio-9dafb28beb71f71df623cf2e09762add40d3009957e80bb00859e4d27c9835d3 WatchSource:0}: Error finding container 9dafb28beb71f71df623cf2e09762add40d3009957e80bb00859e4d27c9835d3: Status 404 returned error can't find the container with id 9dafb28beb71f71df623cf2e09762add40d3009957e80bb00859e4d27c9835d3
Aug 04 03:09:32.077010 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.076949    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tqvgl"
Aug 04 03:09:32.081533 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:32.081492    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a54985_bd71_4de1_83d7_ef1f131cff79.slice/crio-4521cda4b19c9435df524640fb6625fb43734b0ef0f92274d01fe2fcbab1d50f WatchSource:0}: Error finding container 4521cda4b19c9435df524640fb6625fb43734b0ef0f92274d01fe2fcbab1d50f: Status 404 returned error can't find the container with id 4521cda4b19c9435df524640fb6625fb43734b0ef0f92274d01fe2fcbab1d50f
Aug 04 03:09:32.088829 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:09:32.088801    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7cc9c7_8f9d_4a3e_a28d_84d9e599b4d9.slice/crio-42cfb5d3b60b649c4e8719153acb7f84c7d3b8386ac995e4ec0462d5855d5796 WatchSource:0}: Error finding container 42cfb5d3b60b649c4e8719153acb7f84c7d3b8386ac995e4ec0462d5855d5796: Status 404 returned error can't find the container with id 42cfb5d3b60b649c4e8719153acb7f84c7d3b8386ac995e4ec0462d5855d5796
Aug 04 03:09:32.212196 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.212124    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 03:09:32.309683 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.309643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs\") pod \"network-metrics-daemon-c5d74\" (UID: \"b39560be-b9af-4ff7-8a5b-7882c66a03d8\") " pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:32.309831 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:32.309810    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 03:09:32.309907 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:32.309894    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs podName:b39560be-b9af-4ff7-8a5b-7882c66a03d8 nodeName:}" failed. No retries permitted until 2026-08-04 03:09:33.309872831 +0000 UTC m=+3.989660083 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs") pod "network-metrics-daemon-c5d74" (UID: "b39560be-b9af-4ff7-8a5b-7882c66a03d8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 03:09:32.410545 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.410513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret\") pod \"global-pull-secret-syncer-gcg5l\" (UID: \"54b8b42f-6de1-4f5a-8115-50f8b2ff573f\") " pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:32.410745 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:32.410705    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 03:09:32.410904 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:32.410799    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret podName:54b8b42f-6de1-4f5a-8115-50f8b2ff573f nodeName:}" failed. No retries permitted until 2026-08-04 03:09:33.410782503 +0000 UTC m=+4.090569606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret") pod "global-pull-secret-syncer-gcg5l" (UID: "54b8b42f-6de1-4f5a-8115-50f8b2ff573f") : object "kube-system"/"original-pull-secret" not registered
Aug 04 03:09:32.511438 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.511345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vvqt\" (UniqueName: \"kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt\") pod \"network-check-target-qnvmg\" (UID: \"b2d2669b-897a-4fdf-8df5-0cd56e7d47a5\") " pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:32.511571 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:32.511516    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 03:09:32.511571 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:32.511538    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 03:09:32.511571 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:32.511551    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4vvqt for pod openshift-network-diagnostics/network-check-target-qnvmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 03:09:32.511696 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:32.511608    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt podName:b2d2669b-897a-4fdf-8df5-0cd56e7d47a5 nodeName:}" failed. No retries permitted until 2026-08-04 03:09:33.511593541 +0000 UTC m=+4.191380639 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4vvqt" (UniqueName: "kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt") pod "network-check-target-qnvmg" (UID: "b2d2669b-897a-4fdf-8df5-0cd56e7d47a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 03:09:32.732751 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.732715    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 03:04:31 +0000 UTC" deadline="2028-03-30 10:45:40.56567277 +0000 UTC"
Aug 04 03:09:32.732751 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.732747    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14503h36m7.832929562s"
Aug 04 03:09:32.850381 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.850227    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gnp8m" event={"ID":"5f76ca57-4490-4583-85a2-95198fb43ffd","Type":"ContainerStarted","Data":"afeb41c6a0129c7b56c0676f5fd51805f467e3032ddcc0d9be5890d783619f19"}
Aug 04 03:09:32.851730 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.851677    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-57qsq" event={"ID":"d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8","Type":"ContainerStarted","Data":"e53bef1b21d916d8c68e82bce43a2ed48ac0bc6de3c5ce9cbcccf4b205a88490"}
Aug 04 03:09:32.853288 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.853223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v6hhn" event={"ID":"cb1dbb4a-c31f-4485-b847-9a54379f39d1","Type":"ContainerStarted","Data":"9dafb28beb71f71df623cf2e09762add40d3009957e80bb00859e4d27c9835d3"}
Aug 04 03:09:32.854747 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.854692    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zr5zh" event={"ID":"ff980ac9-f6a3-42f9-b1e9-36419f3d927a","Type":"ContainerStarted","Data":"03449524e82c8620a4193b8f61c290d9c9ca78a4229ffffec8c65f414f6d8116"}
Aug 04 03:09:32.856428 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.856386    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbbch" event={"ID":"2c429971-1d7d-4d34-b0fb-c2081a928855","Type":"ContainerStarted","Data":"c498609a7de8f8851e644b6d9c3f5ebedbd55a04442f7897ac288c91db8b1450"}
Aug 04 03:09:32.857892 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.857862    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh" event={"ID":"7df4e9d6-1779-4898-a32b-3b6e7d569896","Type":"ContainerStarted","Data":"0fc7e2b51fad0799f1c9d9cb360a8809f2c5093501f1d41d813850f22169314f"}
Aug 04 03:09:32.859580 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.859506    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tqvgl" event={"ID":"dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9","Type":"ContainerStarted","Data":"42cfb5d3b60b649c4e8719153acb7f84c7d3b8386ac995e4ec0462d5855d5796"}
Aug 04 03:09:32.861183 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.861112    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d" event={"ID":"59a54985-bd71-4de1-83d7-ef1f131cff79","Type":"ContainerStarted","Data":"4521cda4b19c9435df524640fb6625fb43734b0ef0f92274d01fe2fcbab1d50f"}
Aug 04 03:09:32.862585 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:32.862566    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h5jxp" event={"ID":"fa970b56-8654-40d5-b836-005f3c50b89f","Type":"ContainerStarted","Data":"ce22dff1d85baed9a70d5f06b4722804ae344a303e090a3da8a211595e61d700"}
Aug 04 03:09:33.319371 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:33.319294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs\") pod \"network-metrics-daemon-c5d74\" (UID: \"b39560be-b9af-4ff7-8a5b-7882c66a03d8\") " pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:33.319524 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:33.319480    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 03:09:33.319594 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:33.319565    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs podName:b39560be-b9af-4ff7-8a5b-7882c66a03d8 nodeName:}" failed. No retries permitted until 2026-08-04 03:09:35.319542493 +0000 UTC m=+5.999329611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs") pod "network-metrics-daemon-c5d74" (UID: "b39560be-b9af-4ff7-8a5b-7882c66a03d8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 03:09:33.420080 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:33.419824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret\") pod \"global-pull-secret-syncer-gcg5l\" (UID: \"54b8b42f-6de1-4f5a-8115-50f8b2ff573f\") " pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:33.420255 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:33.419988    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 03:09:33.420329 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:33.420309    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret podName:54b8b42f-6de1-4f5a-8115-50f8b2ff573f nodeName:}" failed. No retries permitted until 2026-08-04 03:09:35.420289312 +0000 UTC m=+6.100076434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret") pod "global-pull-secret-syncer-gcg5l" (UID: "54b8b42f-6de1-4f5a-8115-50f8b2ff573f") : object "kube-system"/"original-pull-secret" not registered
Aug 04 03:09:33.521252 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:33.521213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vvqt\" (UniqueName: \"kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt\") pod \"network-check-target-qnvmg\" (UID: \"b2d2669b-897a-4fdf-8df5-0cd56e7d47a5\") " pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:33.521410 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:33.521400    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 03:09:33.521489 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:33.521418    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 03:09:33.521489 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:33.521432    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4vvqt for pod openshift-network-diagnostics/network-check-target-qnvmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 03:09:33.521911 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:33.521846    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt podName:b2d2669b-897a-4fdf-8df5-0cd56e7d47a5 nodeName:}" failed. No retries permitted until 2026-08-04 03:09:35.5214948 +0000 UTC m=+6.201281903 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4vvqt" (UniqueName: "kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt") pod "network-check-target-qnvmg" (UID: "b2d2669b-897a-4fdf-8df5-0cd56e7d47a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 03:09:33.733772 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:33.733633    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 03:04:31 +0000 UTC" deadline="2028-03-18 09:47:54.397897098 +0000 UTC"
Aug 04 03:09:33.733772 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:33.733697    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14214h38m20.664204648s"
Aug 04 03:09:33.841390 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:33.841273    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:33.841474 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:33.841404    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:09:33.841783 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:33.841764    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:33.841873 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:33.841857    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcg5l" podUID="54b8b42f-6de1-4f5a-8115-50f8b2ff573f"
Aug 04 03:09:33.841932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:33.841915    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:33.842041 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:33.842024    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnvmg" podUID="b2d2669b-897a-4fdf-8df5-0cd56e7d47a5"
Aug 04 03:09:33.871792 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:33.871158    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal" event={"ID":"3fb028bcdaa2f5ddcb427d9e021fdc0b","Type":"ContainerStarted","Data":"2408076efbd4b2f5b7d2c40da38bf906b955385d92e5900775609b5159bd7f24"}
Aug 04 03:09:33.886251 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:33.885338    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal" podStartSLOduration=2.885317176 podStartE2EDuration="2.885317176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:09:33.884915661 +0000 UTC m=+4.564702802" watchObservedRunningTime="2026-08-04 03:09:33.885317176 +0000 UTC m=+4.565104297"
Aug 04 03:09:34.887602 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:34.887562    2579 generic.go:356] "Generic (PLEG): container finished" podID="c3e36ae9120ed68114f2482f70b5b344" containerID="7034fd7e1c8ae3c40657d9b54c80f338476814350a6fc576d739e5e206e43c16" exitCode=0
Aug 04 03:09:34.888078 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:34.887709    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal" event={"ID":"c3e36ae9120ed68114f2482f70b5b344","Type":"ContainerDied","Data":"7034fd7e1c8ae3c40657d9b54c80f338476814350a6fc576d739e5e206e43c16"}
Aug 04 03:09:35.338179 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:35.337710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs\") pod \"network-metrics-daemon-c5d74\" (UID: \"b39560be-b9af-4ff7-8a5b-7882c66a03d8\") " pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:35.338179 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:35.337899    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 03:09:35.338179 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:35.337978    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs podName:b39560be-b9af-4ff7-8a5b-7882c66a03d8 nodeName:}" failed. No retries permitted until 2026-08-04 03:09:39.337946007 +0000 UTC m=+10.017733106 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs") pod "network-metrics-daemon-c5d74" (UID: "b39560be-b9af-4ff7-8a5b-7882c66a03d8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 03:09:35.439037 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:35.438296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret\") pod \"global-pull-secret-syncer-gcg5l\" (UID: \"54b8b42f-6de1-4f5a-8115-50f8b2ff573f\") " pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:35.439037 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:35.438494    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 03:09:35.439037 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:35.438556    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret podName:54b8b42f-6de1-4f5a-8115-50f8b2ff573f nodeName:}" failed. No retries permitted until 2026-08-04 03:09:39.438536057 +0000 UTC m=+10.118323162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret") pod "global-pull-secret-syncer-gcg5l" (UID: "54b8b42f-6de1-4f5a-8115-50f8b2ff573f") : object "kube-system"/"original-pull-secret" not registered
Aug 04 03:09:35.539107 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:35.538978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vvqt\" (UniqueName: \"kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt\") pod \"network-check-target-qnvmg\" (UID: \"b2d2669b-897a-4fdf-8df5-0cd56e7d47a5\") " pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:35.539926 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:35.539505    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 03:09:35.539926 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:35.539529    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 03:09:35.539926 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:35.539542    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4vvqt for pod openshift-network-diagnostics/network-check-target-qnvmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 03:09:35.539926 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:35.539606    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt podName:b2d2669b-897a-4fdf-8df5-0cd56e7d47a5 nodeName:}" failed. No retries permitted until 2026-08-04 03:09:39.539587241 +0000 UTC m=+10.219374350 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4vvqt" (UniqueName: "kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt") pod "network-check-target-qnvmg" (UID: "b2d2669b-897a-4fdf-8df5-0cd56e7d47a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 03:09:35.838774 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:35.838747    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:35.838939 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:35.838747    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:35.838939 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:35.838890    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:09:35.839059 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:35.838942    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcg5l" podUID="54b8b42f-6de1-4f5a-8115-50f8b2ff573f"
Aug 04 03:09:35.839059 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:35.839017    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:35.839167 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:35.839088    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnvmg" podUID="b2d2669b-897a-4fdf-8df5-0cd56e7d47a5"
Aug 04 03:09:37.838925 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:37.838890    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:37.839392 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:37.839047    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcg5l" podUID="54b8b42f-6de1-4f5a-8115-50f8b2ff573f"
Aug 04 03:09:37.839470 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:37.839417    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:37.839525 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:37.839505    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnvmg" podUID="b2d2669b-897a-4fdf-8df5-0cd56e7d47a5"
Aug 04 03:09:37.839582 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:37.839559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:37.840199 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:37.839636    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:09:39.373543 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:39.373447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs\") pod \"network-metrics-daemon-c5d74\" (UID: \"b39560be-b9af-4ff7-8a5b-7882c66a03d8\") " pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:39.374034 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:39.373632    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 03:09:39.374034 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:39.373736    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs podName:b39560be-b9af-4ff7-8a5b-7882c66a03d8 nodeName:}" failed. No retries permitted until 2026-08-04 03:09:47.373713915 +0000 UTC m=+18.053501017 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs") pod "network-metrics-daemon-c5d74" (UID: "b39560be-b9af-4ff7-8a5b-7882c66a03d8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 03:09:39.474892 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:39.474288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret\") pod \"global-pull-secret-syncer-gcg5l\" (UID: \"54b8b42f-6de1-4f5a-8115-50f8b2ff573f\") " pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:39.474892 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:39.474493    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 03:09:39.474892 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:39.474558    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret podName:54b8b42f-6de1-4f5a-8115-50f8b2ff573f nodeName:}" failed. No retries permitted until 2026-08-04 03:09:47.474539245 +0000 UTC m=+18.154326369 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret") pod "global-pull-secret-syncer-gcg5l" (UID: "54b8b42f-6de1-4f5a-8115-50f8b2ff573f") : object "kube-system"/"original-pull-secret" not registered
Aug 04 03:09:39.575622 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:39.575578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vvqt\" (UniqueName: \"kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt\") pod \"network-check-target-qnvmg\" (UID: \"b2d2669b-897a-4fdf-8df5-0cd56e7d47a5\") " pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:39.575801 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:39.575770    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 03:09:39.575801 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:39.575797    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 03:09:39.575906 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:39.575811    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4vvqt for pod openshift-network-diagnostics/network-check-target-qnvmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 03:09:39.575906 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:39.575871    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt podName:b2d2669b-897a-4fdf-8df5-0cd56e7d47a5 nodeName:}" failed. No retries permitted until 2026-08-04 03:09:47.575851828 +0000 UTC m=+18.255638940 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4vvqt" (UniqueName: "kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt") pod "network-check-target-qnvmg" (UID: "b2d2669b-897a-4fdf-8df5-0cd56e7d47a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 03:09:39.839762 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:39.839407    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:39.839762 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:39.839517    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:39.839762 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:39.839572    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:09:39.839762 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:39.839592    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcg5l" podUID="54b8b42f-6de1-4f5a-8115-50f8b2ff573f"
Aug 04 03:09:39.839762 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:39.839618    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:39.839762 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:39.839683    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnvmg" podUID="b2d2669b-897a-4fdf-8df5-0cd56e7d47a5"
Aug 04 03:09:41.839013 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:41.838724    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:41.839013 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:41.838864    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcg5l" podUID="54b8b42f-6de1-4f5a-8115-50f8b2ff573f"
Aug 04 03:09:41.839013 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:41.838724    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:41.839013 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:41.838950    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnvmg" podUID="b2d2669b-897a-4fdf-8df5-0cd56e7d47a5"
Aug 04 03:09:41.839013 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:41.838724    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:41.839614 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:41.839048    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:09:43.839360 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:43.839325    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:43.839875 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:43.839325    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:43.839875 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:43.839456    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:09:43.839875 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:43.839536    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcg5l" podUID="54b8b42f-6de1-4f5a-8115-50f8b2ff573f"
Aug 04 03:09:43.839875 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:43.839325    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:43.839875 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:43.839650    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnvmg" podUID="b2d2669b-897a-4fdf-8df5-0cd56e7d47a5"
Aug 04 03:09:45.838836 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:45.838806    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:45.838836 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:45.838837    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:45.839387 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:45.838806    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:45.839387 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:45.838922    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnvmg" podUID="b2d2669b-897a-4fdf-8df5-0cd56e7d47a5"
Aug 04 03:09:45.839387 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:45.839012    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcg5l" podUID="54b8b42f-6de1-4f5a-8115-50f8b2ff573f"
Aug 04 03:09:45.839387 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:45.839105    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:09:47.436424 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:47.436383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs\") pod \"network-metrics-daemon-c5d74\" (UID: \"b39560be-b9af-4ff7-8a5b-7882c66a03d8\") " pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:47.437028 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:47.436570    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 03:09:47.437028 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:47.436655    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs podName:b39560be-b9af-4ff7-8a5b-7882c66a03d8 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:03.436633665 +0000 UTC m=+34.116420768 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs") pod "network-metrics-daemon-c5d74" (UID: "b39560be-b9af-4ff7-8a5b-7882c66a03d8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 03:09:47.536986 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:47.536934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret\") pod \"global-pull-secret-syncer-gcg5l\" (UID: \"54b8b42f-6de1-4f5a-8115-50f8b2ff573f\") " pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:47.537159 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:47.537109    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 03:09:47.537210 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:47.537185    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret podName:54b8b42f-6de1-4f5a-8115-50f8b2ff573f nodeName:}" failed. No retries permitted until 2026-08-04 03:10:03.537166792 +0000 UTC m=+34.216953902 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret") pod "global-pull-secret-syncer-gcg5l" (UID: "54b8b42f-6de1-4f5a-8115-50f8b2ff573f") : object "kube-system"/"original-pull-secret" not registered
Aug 04 03:09:47.637629 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:47.637589    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vvqt\" (UniqueName: \"kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt\") pod \"network-check-target-qnvmg\" (UID: \"b2d2669b-897a-4fdf-8df5-0cd56e7d47a5\") " pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:47.637816 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:47.637747    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 03:09:47.637816 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:47.637773    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 03:09:47.637816 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:47.637786    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4vvqt for pod openshift-network-diagnostics/network-check-target-qnvmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 03:09:47.637995 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:47.637850    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt podName:b2d2669b-897a-4fdf-8df5-0cd56e7d47a5 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:03.637829863 +0000 UTC m=+34.317616965 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4vvqt" (UniqueName: "kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt") pod "network-check-target-qnvmg" (UID: "b2d2669b-897a-4fdf-8df5-0cd56e7d47a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 03:09:47.838906 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:47.838819    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:47.839082 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:47.838836    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:47.839082 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:47.838943    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:47.839082 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:47.838950    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnvmg" podUID="b2d2669b-897a-4fdf-8df5-0cd56e7d47a5"
Aug 04 03:09:47.839082 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:47.839060    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:09:47.839282 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:47.839136    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcg5l" podUID="54b8b42f-6de1-4f5a-8115-50f8b2ff573f"
Aug 04 03:09:49.840455 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:49.840421    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:49.840890 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:49.840545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:49.840890 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:49.840563    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcg5l" podUID="54b8b42f-6de1-4f5a-8115-50f8b2ff573f"
Aug 04 03:09:49.840890 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:49.840663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:49.840890 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:49.840708    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:09:49.840890 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:49.840745    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnvmg" podUID="b2d2669b-897a-4fdf-8df5-0cd56e7d47a5"
Aug 04 03:09:51.839281 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:51.838978    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:51.839843 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:51.838984    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:51.839843 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:51.839363    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcg5l" podUID="54b8b42f-6de1-4f5a-8115-50f8b2ff573f"
Aug 04 03:09:51.839843 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:51.839464    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:09:51.839843 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:51.838984    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:51.839843 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:51.839567    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnvmg" podUID="b2d2669b-897a-4fdf-8df5-0cd56e7d47a5"
Aug 04 03:09:51.922270 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:51.922238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zr5zh" event={"ID":"ff980ac9-f6a3-42f9-b1e9-36419f3d927a","Type":"ContainerStarted","Data":"b2810d8300674076186c6b4d866b23328349128417169342d0f2cbe9ae6c1e7d"}
Aug 04 03:09:51.923742 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:51.923720    2579 generic.go:356] "Generic (PLEG): container finished" podID="2c429971-1d7d-4d34-b0fb-c2081a928855" containerID="f430dccadd6fc400f3e1119a307b7fa832d973850856b5427a082b772c1a89eb" exitCode=0
Aug 04 03:09:51.923840 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:51.923804    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbbch" event={"ID":"2c429971-1d7d-4d34-b0fb-c2081a928855","Type":"ContainerDied","Data":"f430dccadd6fc400f3e1119a307b7fa832d973850856b5427a082b772c1a89eb"}
Aug 04 03:09:51.926330 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:51.926305    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh" event={"ID":"7df4e9d6-1779-4898-a32b-3b6e7d569896","Type":"ContainerStarted","Data":"c88481d6b0665cac10fce0b65f6505e49af54b1a5abebeb7c1d7d7726ed2565c"}
Aug 04 03:09:51.926418 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:51.926381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh" event={"ID":"7df4e9d6-1779-4898-a32b-3b6e7d569896","Type":"ContainerStarted","Data":"505c6d71e548c08a50bcb7a2b0c591fee5608f295c6bf679471d509509f4d817"}
Aug 04 03:09:51.926495 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:51.926435    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh" event={"ID":"7df4e9d6-1779-4898-a32b-3b6e7d569896","Type":"ContainerStarted","Data":"a742f9c6c80f754f5d88afcda640a2b83dd5a9675e0b2254a8ea529eac2d62b7"}
Aug 04 03:09:51.926495 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:51.926485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh" event={"ID":"7df4e9d6-1779-4898-a32b-3b6e7d569896","Type":"ContainerStarted","Data":"9669fa12df1aff6ddf4367883cd947f88863deda5c468b3c0f5517ea293bf868"}
Aug 04 03:09:51.927607 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:51.927588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d" event={"ID":"59a54985-bd71-4de1-83d7-ef1f131cff79","Type":"ContainerStarted","Data":"08c5096ea8cf48218e2982ad31817b164ea001de5b9716da0d60d67e16ac390c"}
Aug 04 03:09:51.928776 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:51.928752    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h5jxp" event={"ID":"fa970b56-8654-40d5-b836-005f3c50b89f","Type":"ContainerStarted","Data":"2dd58296adade114c605505722df8c9e4d24d0d5a65f9e73dbc7374b45364b93"}
Aug 04 03:09:51.930362 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:51.930345    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gnp8m" event={"ID":"5f76ca57-4490-4583-85a2-95198fb43ffd","Type":"ContainerStarted","Data":"751a37b3b88246641b2ad83f7fc889c5fd934412dc1f34d0f6801952ece88755"}
Aug 04 03:09:51.931601 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:51.931581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-57qsq" event={"ID":"d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8","Type":"ContainerStarted","Data":"d80fa90485e9d036723ad24e1eafa52600acea7cc456be860750a44c2354993f"}
Aug 04 03:09:51.933249 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:51.933150    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal" event={"ID":"c3e36ae9120ed68114f2482f70b5b344","Type":"ContainerStarted","Data":"32afcd1d7e0775290c892ec5439a0641a46055ce55e5783ce584e87bef5f9d91"}
Aug 04 03:09:51.934566 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:51.934546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v6hhn" event={"ID":"cb1dbb4a-c31f-4485-b847-9a54379f39d1","Type":"ContainerStarted","Data":"aca6f116ca38103a975e4d061a9439c665982238b63996e445662733861c1d67"}
Aug 04 03:09:51.938138 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:51.938104    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zr5zh" podStartSLOduration=2.630694634 podStartE2EDuration="21.938091767s" totalImagesPullingTime="19.307397133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:09:51.93769712 +0000 UTC m=+22.617484240" watchObservedRunningTime="2026-08-04 03:09:51.938091767 +0000 UTC m=+22.617878886"
Aug 04 03:09:51.950260 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:51.950214    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v6hhn" podStartSLOduration=2.675265284 podStartE2EDuration="21.950201037s" totalImagesPullingTime="19.274935753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:09:51.949819369 +0000 UTC m=+22.629606489" watchObservedRunningTime="2026-08-04 03:09:51.950201037 +0000 UTC m=+22.629988187"
Aug 04 03:09:51.976448 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:51.976410    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal" podStartSLOduration=20.976395519 podStartE2EDuration="20.976395519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:09:51.973659081 +0000 UTC m=+22.653446200" watchObservedRunningTime="2026-08-04 03:09:51.976395519 +0000 UTC m=+22.656182638"
Aug 04 03:09:52.041890 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:52.041850    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gnp8m" podStartSLOduration=2.735124865 podStartE2EDuration="22.041834629s" totalImagesPullingTime="19.306709764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:09:52.017732431 +0000 UTC m=+22.697519550" watchObservedRunningTime="2026-08-04 03:09:52.041834629 +0000 UTC m=+22.721621751"
Aug 04 03:09:52.042093 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:52.042063    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-h5jxp" podStartSLOduration=2.715348011 podStartE2EDuration="22.042051565s" totalImagesPullingTime="19.326703554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:09:52.041234661 +0000 UTC m=+22.721021791" watchObservedRunningTime="2026-08-04 03:09:52.042051565 +0000 UTC m=+22.721838685"
Aug 04 03:09:52.056939 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:52.056899    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-57qsq" podStartSLOduration=2.70634401 podStartE2EDuration="22.056885095s" totalImagesPullingTime="19.350541085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:09:52.056567585 +0000 UTC m=+22.736354704" watchObservedRunningTime="2026-08-04 03:09:52.056885095 +0000 UTC m=+22.736672214"
Aug 04 03:09:52.321124 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:52.321038    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v6hhn"
Aug 04 03:09:52.940241 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:52.940191    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh" event={"ID":"7df4e9d6-1779-4898-a32b-3b6e7d569896","Type":"ContainerStarted","Data":"541aa46d2f7af57b87289c2f2741fe8452b924e9e4597b2dcdb229baf207c53e"}
Aug 04 03:09:52.940800 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:52.940329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh" event={"ID":"7df4e9d6-1779-4898-a32b-3b6e7d569896","Type":"ContainerStarted","Data":"301471ceee397c7ee216e62978a4b03749ca929248aa65a3919050f3b4159b66"}
Aug 04 03:09:52.941800 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:52.941742    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tqvgl" event={"ID":"dc7cc9c7-8f9d-4a3e-a28d-84d9e599b4d9","Type":"ContainerStarted","Data":"686606b58f7c4eaf60e0dfc504ad0227d6ee635442c1606ec23f90b321bb9370"}
Aug 04 03:09:53.223271 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:53.223184    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v6hhn"
Aug 04 03:09:53.224007 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:53.223984    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v6hhn"
Aug 04 03:09:53.243718 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:53.243664    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tqvgl" podStartSLOduration=3.9850185700000003 podStartE2EDuration="23.243641744s" totalImagesPullingTime="19.258623174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:09:52.955538571 +0000 UTC m=+23.635325692" watchObservedRunningTime="2026-08-04 03:09:53.243641744 +0000 UTC m=+23.923428864"
Aug 04 03:09:53.416145 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:53.416120    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Aug 04 03:09:53.761322 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:53.761134    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T03:09:53.416138865Z","UUID":"2b62b8d6-04f9-4b00-a83c-40023e4b47b8","Handler":null,"Name":"","Endpoint":""}
Aug 04 03:09:53.762800 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:53.762772    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Aug 04 03:09:53.762943 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:53.762813    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Aug 04 03:09:53.838977 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:53.838936    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:53.839142 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:53.838939    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:53.839142 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:53.839064    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcg5l" podUID="54b8b42f-6de1-4f5a-8115-50f8b2ff573f"
Aug 04 03:09:53.839142 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:53.839073    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:53.839358 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:53.839152    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:09:53.839358 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:53.839222    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnvmg" podUID="b2d2669b-897a-4fdf-8df5-0cd56e7d47a5"
Aug 04 03:09:53.946773 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:53.946737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d" event={"ID":"59a54985-bd71-4de1-83d7-ef1f131cff79","Type":"ContainerStarted","Data":"b2b446b29ebc6b36a22e80357967e5533af33ab36fabe814de648f26380720e4"}
Aug 04 03:09:53.947562 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:53.947542    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v6hhn"
Aug 04 03:09:54.954293 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:54.954256    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh" event={"ID":"7df4e9d6-1779-4898-a32b-3b6e7d569896","Type":"ContainerStarted","Data":"8aaa8fc0aef46fce5afdd66455bba8dd0ae9784142989b8ef92924a871443851"}
Aug 04 03:09:55.839701 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:55.839455    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:55.839889 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:55.839456    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:55.839889 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:55.839811    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcg5l" podUID="54b8b42f-6de1-4f5a-8115-50f8b2ff573f"
Aug 04 03:09:55.839889 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:55.839878    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnvmg" podUID="b2d2669b-897a-4fdf-8df5-0cd56e7d47a5"
Aug 04 03:09:55.840064 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:55.839483    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:55.840064 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:55.839978    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:09:56.960886 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:56.960647    2579 generic.go:356] "Generic (PLEG): container finished" podID="2c429971-1d7d-4d34-b0fb-c2081a928855" containerID="f57aae1c8e2e468767b692dcbf5b9651696cecda1a5beef54e34bb02703c6bed" exitCode=0
Aug 04 03:09:56.961755 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:56.960706    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbbch" event={"ID":"2c429971-1d7d-4d34-b0fb-c2081a928855","Type":"ContainerDied","Data":"f57aae1c8e2e468767b692dcbf5b9651696cecda1a5beef54e34bb02703c6bed"}
Aug 04 03:09:56.964644 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:56.964617    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh" event={"ID":"7df4e9d6-1779-4898-a32b-3b6e7d569896","Type":"ContainerStarted","Data":"707dced8fb83b390601a0a4975097bdfdf48f97ce54a473820c2e35e584cb453"}
Aug 04 03:09:56.964833 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:56.964816    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:56.966752 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:56.966722    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d" event={"ID":"59a54985-bd71-4de1-83d7-ef1f131cff79","Type":"ContainerStarted","Data":"2c1c3e790d1f608b03898aabd125840f00954787b7b67bcb32e6a1ab57ea65ab"}
Aug 04 03:09:56.980321 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:56.980296    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:56.996970 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:56.996909    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vd42d" podStartSLOduration=3.155220227 podStartE2EDuration="26.996895493s" totalImagesPullingTime="23.841675266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:09:56.996691134 +0000 UTC m=+27.676478252" watchObservedRunningTime="2026-08-04 03:09:56.996895493 +0000 UTC m=+27.676682612"
Aug 04 03:09:57.838978 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:57.838931    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:57.839199 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:57.838996    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:57.839199 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:57.838937    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:57.839199 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:57.839072    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcg5l" podUID="54b8b42f-6de1-4f5a-8115-50f8b2ff573f"
Aug 04 03:09:57.839199 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:57.839120    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnvmg" podUID="b2d2669b-897a-4fdf-8df5-0cd56e7d47a5"
Aug 04 03:09:57.839381 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:57.839200    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:09:57.969843 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:57.969815    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:57.970261 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:57.969895    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:57.985359 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:57.985328    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:09:58.014183 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:58.014135    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh" podStartSLOduration=8.518667749 podStartE2EDuration="28.014120053s" totalImagesPullingTime="19.495452304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:09:57.022656788 +0000 UTC m=+27.702443928" watchObservedRunningTime="2026-08-04 03:09:58.014120053 +0000 UTC m=+28.693907171"
Aug 04 03:09:58.445551 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:58.445469    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c5d74"]
Aug 04 03:09:58.445725 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:58.445635    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:58.445774 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:58.445743    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:09:58.448616 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:58.448588    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qnvmg"]
Aug 04 03:09:58.448753 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:58.448700    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:58.448816 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:58.448797    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnvmg" podUID="b2d2669b-897a-4fdf-8df5-0cd56e7d47a5"
Aug 04 03:09:58.449423 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:58.449399    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gcg5l"]
Aug 04 03:09:58.449524 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:58.449510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:58.449629 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:58.449612    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcg5l" podUID="54b8b42f-6de1-4f5a-8115-50f8b2ff573f"
Aug 04 03:09:58.974329 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:58.974294    2579 generic.go:356] "Generic (PLEG): container finished" podID="2c429971-1d7d-4d34-b0fb-c2081a928855" containerID="74e4a4479297ca97b8b8c8beea15f7985598452beccdc8440ab6fc07e653ca06" exitCode=0
Aug 04 03:09:58.974783 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:58.974344    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbbch" event={"ID":"2c429971-1d7d-4d34-b0fb-c2081a928855","Type":"ContainerDied","Data":"74e4a4479297ca97b8b8c8beea15f7985598452beccdc8440ab6fc07e653ca06"}
Aug 04 03:09:59.840054 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:59.840020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:09:59.840216 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:59.840112    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcg5l" podUID="54b8b42f-6de1-4f5a-8115-50f8b2ff573f"
Aug 04 03:09:59.840216 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:59.840199    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:09:59.840341 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:59.840324    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:09:59.840395 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:09:59.840359    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:09:59.840444 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:09:59.840432    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnvmg" podUID="b2d2669b-897a-4fdf-8df5-0cd56e7d47a5"
Aug 04 03:10:00.981223 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:00.981186    2579 generic.go:356] "Generic (PLEG): container finished" podID="2c429971-1d7d-4d34-b0fb-c2081a928855" containerID="f95bb45a403ce8762cbbb56de5d0f3359364e70ff38d0a029e262c5bcf090b3a" exitCode=0
Aug 04 03:10:00.981754 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:00.981238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbbch" event={"ID":"2c429971-1d7d-4d34-b0fb-c2081a928855","Type":"ContainerDied","Data":"f95bb45a403ce8762cbbb56de5d0f3359364e70ff38d0a029e262c5bcf090b3a"}
Aug 04 03:10:01.839203 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:01.838940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:10:01.839378 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:01.838943    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:10:01.839378 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:01.839312    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcg5l" podUID="54b8b42f-6de1-4f5a-8115-50f8b2ff573f"
Aug 04 03:10:01.839508 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:01.838947    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:10:01.839508 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:01.839425    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:10:01.839508 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:01.839467    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnvmg" podUID="b2d2669b-897a-4fdf-8df5-0cd56e7d47a5"
Aug 04 03:10:03.451505 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:03.451461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs\") pod \"network-metrics-daemon-c5d74\" (UID: \"b39560be-b9af-4ff7-8a5b-7882c66a03d8\") " pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:10:03.451880 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:03.451581    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 03:10:03.451880 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:03.451644    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs podName:b39560be-b9af-4ff7-8a5b-7882c66a03d8 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:35.45162509 +0000 UTC m=+66.131412186 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs") pod "network-metrics-daemon-c5d74" (UID: "b39560be-b9af-4ff7-8a5b-7882c66a03d8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 03:10:03.552791 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:03.552752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret\") pod \"global-pull-secret-syncer-gcg5l\" (UID: \"54b8b42f-6de1-4f5a-8115-50f8b2ff573f\") " pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:10:03.552999 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:03.552933    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 03:10:03.553064 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:03.553032    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret podName:54b8b42f-6de1-4f5a-8115-50f8b2ff573f nodeName:}" failed. No retries permitted until 2026-08-04 03:10:35.553010261 +0000 UTC m=+66.232797362 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret") pod "global-pull-secret-syncer-gcg5l" (UID: "54b8b42f-6de1-4f5a-8115-50f8b2ff573f") : object "kube-system"/"original-pull-secret" not registered
Aug 04 03:10:03.654095 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:03.654051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vvqt\" (UniqueName: \"kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt\") pod \"network-check-target-qnvmg\" (UID: \"b2d2669b-897a-4fdf-8df5-0cd56e7d47a5\") " pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:10:03.654272 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:03.654223    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 03:10:03.654272 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:03.654246    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 03:10:03.654272 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:03.654256    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4vvqt for pod openshift-network-diagnostics/network-check-target-qnvmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 03:10:03.654391 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:03.654307    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt podName:b2d2669b-897a-4fdf-8df5-0cd56e7d47a5 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:35.6542924 +0000 UTC m=+66.334079497 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4vvqt" (UniqueName: "kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt") pod "network-check-target-qnvmg" (UID: "b2d2669b-897a-4fdf-8df5-0cd56e7d47a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 03:10:03.839001 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:03.838954    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:10:03.839196 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:03.838977    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:10:03.839196 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:03.839111    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:10:03.839196 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:03.838954    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:10:03.839366 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:03.839144    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcg5l" podUID="54b8b42f-6de1-4f5a-8115-50f8b2ff573f"
Aug 04 03:10:03.839366 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:03.839268    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnvmg" podUID="b2d2669b-897a-4fdf-8df5-0cd56e7d47a5"
Aug 04 03:10:05.128719 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.128684    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeReady"
Aug 04 03:10:05.129284 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.128854    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Aug 04 03:10:05.162263 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.162229    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"]
Aug 04 03:10:05.166896 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.166871    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-958dbfdbb-wz2mz"]
Aug 04 03:10:05.167045 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.167020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"
Aug 04 03:10:05.169400 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.169378    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-g7q88\""
Aug 04 03:10:05.169400 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.169388    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Aug 04 03:10:05.169581 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.169445    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Aug 04 03:10:05.170345 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.170327    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.172640 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.172623    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9lvw5\""
Aug 04 03:10:05.172895 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.172877    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Aug 04 03:10:05.172895 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.172887    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Aug 04 03:10:05.173055 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.173040    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Aug 04 03:10:05.177926 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.177902    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"]
Aug 04 03:10:05.182293 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.182274    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Aug 04 03:10:05.187721 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.187653    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-958dbfdbb-wz2mz"]
Aug 04 03:10:05.188133 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.188111    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-np56j"]
Aug 04 03:10:05.191800 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.191780    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ztnbv"]
Aug 04 03:10:05.191913 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.191885    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-np56j"
Aug 04 03:10:05.194396 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.194372    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Aug 04 03:10:05.194644 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.194623    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Aug 04 03:10:05.194880 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.194864    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4d7vw\""
Aug 04 03:10:05.195400 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.195379    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ztnbv"
Aug 04 03:10:05.196428 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.196411    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-np56j"]
Aug 04 03:10:05.198388 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.198322    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ztnbv"]
Aug 04 03:10:05.199169 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.199143    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Aug 04 03:10:05.199528 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.199509    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xvw8f\""
Aug 04 03:10:05.199621 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.199515    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Aug 04 03:10:05.200291 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.200274    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Aug 04 03:10:05.265450 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.265413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f607efb-3bb4-4755-a366-4d8b0c5aa431-image-registry-private-configuration\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.265450 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.265462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f607efb-3bb4-4755-a366-4d8b0c5aa431-ca-trust-extracted\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.265701 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.265493    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c78063e-e4f9-446b-90c5-89f0f2ddc001-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-2t9qv\" (UID: \"9c78063e-e4f9-446b-90c5-89f0f2ddc001\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"
Aug 04 03:10:05.265701 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.265522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-2t9qv\" (UID: \"9c78063e-e4f9-446b-90c5-89f0f2ddc001\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"
Aug 04 03:10:05.265701 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.265615    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.265701 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.265649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz6tw\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-kube-api-access-zz6tw\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.265701 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.265696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-bound-sa-token\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.265974 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.265716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f607efb-3bb4-4755-a366-4d8b0c5aa431-installation-pull-secrets\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.265974 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.265794    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-certificates\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.265974 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.265832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f607efb-3bb4-4755-a366-4d8b0c5aa431-trusted-ca\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.366730 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.366688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.366914 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:05.366823    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 03:10:05.366914 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:05.366845    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-958dbfdbb-wz2mz: secret "image-registry-tls" not found
Aug 04 03:10:05.366914 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:05.366911    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls podName:5f607efb-3bb4-4755-a366-4d8b0c5aa431 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:05.86688811 +0000 UTC m=+36.546675208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls") pod "image-registry-958dbfdbb-wz2mz" (UID: "5f607efb-3bb4-4755-a366-4d8b0c5aa431") : secret "image-registry-tls" not found
Aug 04 03:10:05.367138 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.366828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz6tw\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-kube-api-access-zz6tw\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.367138 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.366990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc5bd55b-006c-482c-b501-11c7e4a50629-cert\") pod \"ingress-canary-ztnbv\" (UID: \"fc5bd55b-006c-482c-b501-11c7e4a50629\") " pod="openshift-ingress-canary/ingress-canary-ztnbv"
Aug 04 03:10:05.367138 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.367012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6slrc\" (UniqueName: \"kubernetes.io/projected/fc5bd55b-006c-482c-b501-11c7e4a50629-kube-api-access-6slrc\") pod \"ingress-canary-ztnbv\" (UID: \"fc5bd55b-006c-482c-b501-11c7e4a50629\") " pod="openshift-ingress-canary/ingress-canary-ztnbv"
Aug 04 03:10:05.367138 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.367032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b43cf1f-077d-485b-8a0d-fdec861aff43-config-volume\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:10:05.367138 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.367082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-bound-sa-token\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.367138 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.367101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f607efb-3bb4-4755-a366-4d8b0c5aa431-installation-pull-secrets\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.367138 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.367138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-certificates\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.367456 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.367162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f607efb-3bb4-4755-a366-4d8b0c5aa431-trusted-ca\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.367456 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.367184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:10:05.367456 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.367207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b43cf1f-077d-485b-8a0d-fdec861aff43-tmp-dir\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:10:05.367456 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.367254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f607efb-3bb4-4755-a366-4d8b0c5aa431-image-registry-private-configuration\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.367456 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.367277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f607efb-3bb4-4755-a366-4d8b0c5aa431-ca-trust-extracted\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.367456 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.367301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c78063e-e4f9-446b-90c5-89f0f2ddc001-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-2t9qv\" (UID: \"9c78063e-e4f9-446b-90c5-89f0f2ddc001\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"
Aug 04 03:10:05.367456 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.367325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-2t9qv\" (UID: \"9c78063e-e4f9-446b-90c5-89f0f2ddc001\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"
Aug 04 03:10:05.367456 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.367363    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qt82\" (UniqueName: \"kubernetes.io/projected/7b43cf1f-077d-485b-8a0d-fdec861aff43-kube-api-access-5qt82\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:10:05.367819 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:05.367743    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 03:10:05.367819 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:05.367811    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert podName:9c78063e-e4f9-446b-90c5-89f0f2ddc001 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:05.86779318 +0000 UTC m=+36.547580283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-2t9qv" (UID: "9c78063e-e4f9-446b-90c5-89f0f2ddc001") : secret "networking-console-plugin-cert" not found
Aug 04 03:10:05.368221 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.368200    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f607efb-3bb4-4755-a366-4d8b0c5aa431-ca-trust-extracted\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.368414 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.368395    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c78063e-e4f9-446b-90c5-89f0f2ddc001-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-2t9qv\" (UID: \"9c78063e-e4f9-446b-90c5-89f0f2ddc001\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"
Aug 04 03:10:05.368414 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.368405    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-certificates\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.368803 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.368783    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f607efb-3bb4-4755-a366-4d8b0c5aa431-trusted-ca\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.372177 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.372149    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f607efb-3bb4-4755-a366-4d8b0c5aa431-installation-pull-secrets\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.372292 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.372176    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f607efb-3bb4-4755-a366-4d8b0c5aa431-image-registry-private-configuration\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.377286 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.377240    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-bound-sa-token\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.377394 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.377337    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz6tw\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-kube-api-access-zz6tw\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.468027 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.467984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:10:05.468027 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.468029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b43cf1f-077d-485b-8a0d-fdec861aff43-tmp-dir\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:10:05.468275 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.468088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qt82\" (UniqueName: \"kubernetes.io/projected/7b43cf1f-077d-485b-8a0d-fdec861aff43-kube-api-access-5qt82\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:10:05.468275 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.468155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc5bd55b-006c-482c-b501-11c7e4a50629-cert\") pod \"ingress-canary-ztnbv\" (UID: \"fc5bd55b-006c-482c-b501-11c7e4a50629\") " pod="openshift-ingress-canary/ingress-canary-ztnbv"
Aug 04 03:10:05.468275 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:05.468161    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 03:10:05.468275 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.468177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6slrc\" (UniqueName: \"kubernetes.io/projected/fc5bd55b-006c-482c-b501-11c7e4a50629-kube-api-access-6slrc\") pod \"ingress-canary-ztnbv\" (UID: \"fc5bd55b-006c-482c-b501-11c7e4a50629\") " pod="openshift-ingress-canary/ingress-canary-ztnbv"
Aug 04 03:10:05.468275 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.468201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b43cf1f-077d-485b-8a0d-fdec861aff43-config-volume\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:10:05.468275 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:05.468251    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls podName:7b43cf1f-077d-485b-8a0d-fdec861aff43 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:05.968232939 +0000 UTC m=+36.648020036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls") pod "dns-default-np56j" (UID: "7b43cf1f-077d-485b-8a0d-fdec861aff43") : secret "dns-default-metrics-tls" not found
Aug 04 03:10:05.469320 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.469293    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b43cf1f-077d-485b-8a0d-fdec861aff43-config-volume\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:10:05.469441 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.469413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b43cf1f-077d-485b-8a0d-fdec861aff43-tmp-dir\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:10:05.470977 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.470939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc5bd55b-006c-482c-b501-11c7e4a50629-cert\") pod \"ingress-canary-ztnbv\" (UID: \"fc5bd55b-006c-482c-b501-11c7e4a50629\") " pod="openshift-ingress-canary/ingress-canary-ztnbv"
Aug 04 03:10:05.476822 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.476795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6slrc\" (UniqueName: \"kubernetes.io/projected/fc5bd55b-006c-482c-b501-11c7e4a50629-kube-api-access-6slrc\") pod \"ingress-canary-ztnbv\" (UID: \"fc5bd55b-006c-482c-b501-11c7e4a50629\") " pod="openshift-ingress-canary/ingress-canary-ztnbv"
Aug 04 03:10:05.476945 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.476916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qt82\" (UniqueName: \"kubernetes.io/projected/7b43cf1f-077d-485b-8a0d-fdec861aff43-kube-api-access-5qt82\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:10:05.512347 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.512312    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ztnbv"
Aug 04 03:10:05.841003 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.839637    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:10:05.841003 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.839654    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:10:05.841003 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.839645    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:10:05.843227 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.842534    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 03:10:05.843488 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.843210    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 03:10:05.843631 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.843614    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 03:10:05.843831 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.843815    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6l47c\""
Aug 04 03:10:05.844228 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.844065    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 03:10:05.844228 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.844127    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jb5ws\""
Aug 04 03:10:05.871054 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.871029    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ztnbv"]
Aug 04 03:10:05.871242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.871222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-2t9qv\" (UID: \"9c78063e-e4f9-446b-90c5-89f0f2ddc001\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"
Aug 04 03:10:05.871286 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.871274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:05.871365 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:05.871353    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 03:10:05.871413 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:05.871386    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 03:10:05.871413 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:05.871396    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-958dbfdbb-wz2mz: secret "image-registry-tls" not found
Aug 04 03:10:05.871413 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:05.871410    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert podName:9c78063e-e4f9-446b-90c5-89f0f2ddc001 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:06.871391536 +0000 UTC m=+37.551178634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-2t9qv" (UID: "9c78063e-e4f9-446b-90c5-89f0f2ddc001") : secret "networking-console-plugin-cert" not found
Aug 04 03:10:05.871530 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:05.871431    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls podName:5f607efb-3bb4-4755-a366-4d8b0c5aa431 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:06.871420452 +0000 UTC m=+37.551207549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls") pod "image-registry-958dbfdbb-wz2mz" (UID: "5f607efb-3bb4-4755-a366-4d8b0c5aa431") : secret "image-registry-tls" not found
Aug 04 03:10:05.874805 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:10:05.874783    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5bd55b_006c_482c_b501_11c7e4a50629.slice/crio-c0fc2bdaa0d3bc8afddece8812dcc66cd29d920702f5e399395b51bfc0fc6802 WatchSource:0}: Error finding container c0fc2bdaa0d3bc8afddece8812dcc66cd29d920702f5e399395b51bfc0fc6802: Status 404 returned error can't find the container with id c0fc2bdaa0d3bc8afddece8812dcc66cd29d920702f5e399395b51bfc0fc6802
Aug 04 03:10:05.972213 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.972104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:10:05.972345 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:05.972323    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 03:10:05.972419 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:05.972408    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls podName:7b43cf1f-077d-485b-8a0d-fdec861aff43 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:06.972385329 +0000 UTC m=+37.652172445 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls") pod "dns-default-np56j" (UID: "7b43cf1f-077d-485b-8a0d-fdec861aff43") : secret "dns-default-metrics-tls" not found
Aug 04 03:10:05.995304 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.995271    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ztnbv" event={"ID":"fc5bd55b-006c-482c-b501-11c7e4a50629","Type":"ContainerStarted","Data":"c0fc2bdaa0d3bc8afddece8812dcc66cd29d920702f5e399395b51bfc0fc6802"}
Aug 04 03:10:05.998164 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:05.998130    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbbch" event={"ID":"2c429971-1d7d-4d34-b0fb-c2081a928855","Type":"ContainerStarted","Data":"1c34a2bbc840e4c1680cd551ebfbcad17a81833f5570d02f4fda378b73780dd6"}
Aug 04 03:10:06.880572 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:06.880536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-2t9qv\" (UID: \"9c78063e-e4f9-446b-90c5-89f0f2ddc001\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"
Aug 04 03:10:06.881242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:06.880607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:06.881242 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:06.880705    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 03:10:06.881242 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:06.880726    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 03:10:06.881242 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:06.880739    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-958dbfdbb-wz2mz: secret "image-registry-tls" not found
Aug 04 03:10:06.881242 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:06.880784    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert podName:9c78063e-e4f9-446b-90c5-89f0f2ddc001 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:08.880763326 +0000 UTC m=+39.560550426 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-2t9qv" (UID: "9c78063e-e4f9-446b-90c5-89f0f2ddc001") : secret "networking-console-plugin-cert" not found
Aug 04 03:10:06.881242 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:06.880802    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls podName:5f607efb-3bb4-4755-a366-4d8b0c5aa431 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:08.880794842 +0000 UTC m=+39.560581938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls") pod "image-registry-958dbfdbb-wz2mz" (UID: "5f607efb-3bb4-4755-a366-4d8b0c5aa431") : secret "image-registry-tls" not found
Aug 04 03:10:06.981337 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:06.981300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:10:06.981512 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:06.981455    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 03:10:06.981567 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:06.981533    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls podName:7b43cf1f-077d-485b-8a0d-fdec861aff43 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:08.981512583 +0000 UTC m=+39.661299701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls") pod "dns-default-np56j" (UID: "7b43cf1f-077d-485b-8a0d-fdec861aff43") : secret "dns-default-metrics-tls" not found
Aug 04 03:10:07.003859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:07.003818    2579 generic.go:356] "Generic (PLEG): container finished" podID="2c429971-1d7d-4d34-b0fb-c2081a928855" containerID="1c34a2bbc840e4c1680cd551ebfbcad17a81833f5570d02f4fda378b73780dd6" exitCode=0
Aug 04 03:10:07.003859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:07.003849    2579 generic.go:356] "Generic (PLEG): container finished" podID="2c429971-1d7d-4d34-b0fb-c2081a928855" containerID="aaf7eb00afb4b4b9dababfdbcc3744f8423af23424c9e649a6689f8ce5ff0dd2" exitCode=0
Aug 04 03:10:07.004106 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:07.003899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbbch" event={"ID":"2c429971-1d7d-4d34-b0fb-c2081a928855","Type":"ContainerDied","Data":"1c34a2bbc840e4c1680cd551ebfbcad17a81833f5570d02f4fda378b73780dd6"}
Aug 04 03:10:07.004106 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:07.004018    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbbch" event={"ID":"2c429971-1d7d-4d34-b0fb-c2081a928855","Type":"ContainerDied","Data":"aaf7eb00afb4b4b9dababfdbcc3744f8423af23424c9e649a6689f8ce5ff0dd2"}
Aug 04 03:10:08.010167 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:08.010126    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbbch" event={"ID":"2c429971-1d7d-4d34-b0fb-c2081a928855","Type":"ContainerStarted","Data":"5f6369478e8ff3ffc6188f8b056a8eb9ebe41bed0c8c9005312289f1b53eb1c9"}
Aug 04 03:10:08.032688 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:08.032577    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nbbch" podStartSLOduration=7.364408346 podStartE2EDuration="38.032557167s" totalImagesPullingTime="30.668148821s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:10:08.030790651 +0000 UTC m=+38.710577785" watchObservedRunningTime="2026-08-04 03:10:08.032557167 +0000 UTC m=+38.712344286"
Aug 04 03:10:08.896687 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:08.896506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-2t9qv\" (UID: \"9c78063e-e4f9-446b-90c5-89f0f2ddc001\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"
Aug 04 03:10:08.896687 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:08.896666    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 03:10:08.896914 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:08.896706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:08.896914 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:08.896745    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert podName:9c78063e-e4f9-446b-90c5-89f0f2ddc001 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:12.896725504 +0000 UTC m=+43.576512603 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-2t9qv" (UID: "9c78063e-e4f9-446b-90c5-89f0f2ddc001") : secret "networking-console-plugin-cert" not found
Aug 04 03:10:08.896914 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:08.896782    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 03:10:08.896914 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:08.896792    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-958dbfdbb-wz2mz: secret "image-registry-tls" not found
Aug 04 03:10:08.896914 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:08.896832    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls podName:5f607efb-3bb4-4755-a366-4d8b0c5aa431 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:12.896822007 +0000 UTC m=+43.576609104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls") pod "image-registry-958dbfdbb-wz2mz" (UID: "5f607efb-3bb4-4755-a366-4d8b0c5aa431") : secret "image-registry-tls" not found
Aug 04 03:10:08.998075 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:08.998043    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:10:08.998219 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:08.998148    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 03:10:08.998219 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:08.998199    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls podName:7b43cf1f-077d-485b-8a0d-fdec861aff43 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:12.998186005 +0000 UTC m=+43.677973102 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls") pod "dns-default-np56j" (UID: "7b43cf1f-077d-485b-8a0d-fdec861aff43") : secret "dns-default-metrics-tls" not found
Aug 04 03:10:09.013973 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:09.013937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ztnbv" event={"ID":"fc5bd55b-006c-482c-b501-11c7e4a50629","Type":"ContainerStarted","Data":"6027751f676a597caf15d6f34f55672ed7760f9cafd337cc273a14eadcf741b8"}
Aug 04 03:10:09.029297 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:09.029232    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ztnbv" podStartSLOduration=1.733219156 podStartE2EDuration="4.029214265s" totalImagesPullingTime="2.295995109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:10:09.028372669 +0000 UTC m=+39.708159791" watchObservedRunningTime="2026-08-04 03:10:09.029214265 +0000 UTC m=+39.709001387"
Aug 04 03:10:12.926256 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:12.926209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-2t9qv\" (UID: \"9c78063e-e4f9-446b-90c5-89f0f2ddc001\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"
Aug 04 03:10:12.926646 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:12.926271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:12.926646 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:12.926363    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 03:10:12.926646 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:12.926381    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-958dbfdbb-wz2mz: secret "image-registry-tls" not found
Aug 04 03:10:12.926646 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:12.926425    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls podName:5f607efb-3bb4-4755-a366-4d8b0c5aa431 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:20.92641178 +0000 UTC m=+51.606198878 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls") pod "image-registry-958dbfdbb-wz2mz" (UID: "5f607efb-3bb4-4755-a366-4d8b0c5aa431") : secret "image-registry-tls" not found
Aug 04 03:10:12.926646 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:12.926362    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 03:10:12.926646 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:12.926516    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert podName:9c78063e-e4f9-446b-90c5-89f0f2ddc001 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:20.926498743 +0000 UTC m=+51.606285840 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-2t9qv" (UID: "9c78063e-e4f9-446b-90c5-89f0f2ddc001") : secret "networking-console-plugin-cert" not found
Aug 04 03:10:13.026665 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:13.026637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:10:13.026811 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:13.026792    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 03:10:13.026872 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:13.026861    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls podName:7b43cf1f-077d-485b-8a0d-fdec861aff43 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:21.026844859 +0000 UTC m=+51.706631966 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls") pod "dns-default-np56j" (UID: "7b43cf1f-077d-485b-8a0d-fdec861aff43") : secret "dns-default-metrics-tls" not found
Aug 04 03:10:20.983050 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:20.983007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:20.983548 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:20.983089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-2t9qv\" (UID: \"9c78063e-e4f9-446b-90c5-89f0f2ddc001\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"
Aug 04 03:10:20.983548 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:20.983160    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 03:10:20.983548 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:20.983176    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 03:10:20.983548 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:20.983229    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert podName:9c78063e-e4f9-446b-90c5-89f0f2ddc001 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:36.983215497 +0000 UTC m=+67.663002594 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-2t9qv" (UID: "9c78063e-e4f9-446b-90c5-89f0f2ddc001") : secret "networking-console-plugin-cert" not found
Aug 04 03:10:20.983548 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:20.983178    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-958dbfdbb-wz2mz: secret "image-registry-tls" not found
Aug 04 03:10:20.983548 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:20.983304    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls podName:5f607efb-3bb4-4755-a366-4d8b0c5aa431 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:36.983289489 +0000 UTC m=+67.663076586 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls") pod "image-registry-958dbfdbb-wz2mz" (UID: "5f607efb-3bb4-4755-a366-4d8b0c5aa431") : secret "image-registry-tls" not found
Aug 04 03:10:21.083727 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:21.083692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:10:21.083886 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:21.083813    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 03:10:21.083886 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:21.083876    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls podName:7b43cf1f-077d-485b-8a0d-fdec861aff43 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:37.08385968 +0000 UTC m=+67.763646777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls") pod "dns-default-np56j" (UID: "7b43cf1f-077d-485b-8a0d-fdec861aff43") : secret "dns-default-metrics-tls" not found
Aug 04 03:10:29.999087 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:29.999059    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4zfwh"
Aug 04 03:10:35.483805 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:35.483759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs\") pod \"network-metrics-daemon-c5d74\" (UID: \"b39560be-b9af-4ff7-8a5b-7882c66a03d8\") " pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:10:35.486249 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:35.486231    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 03:10:35.494306 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:35.494282    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Aug 04 03:10:35.494369 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:35.494359    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs podName:b39560be-b9af-4ff7-8a5b-7882c66a03d8 nodeName:}" failed. No retries permitted until 2026-08-04 03:11:39.494343354 +0000 UTC m=+130.174130454 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs") pod "network-metrics-daemon-c5d74" (UID: "b39560be-b9af-4ff7-8a5b-7882c66a03d8") : secret "metrics-daemon-secret" not found
Aug 04 03:10:35.585113 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:35.585081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret\") pod \"global-pull-secret-syncer-gcg5l\" (UID: \"54b8b42f-6de1-4f5a-8115-50f8b2ff573f\") " pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:10:35.587715 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:35.587698    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 03:10:35.599384 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:35.599360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54b8b42f-6de1-4f5a-8115-50f8b2ff573f-original-pull-secret\") pod \"global-pull-secret-syncer-gcg5l\" (UID: \"54b8b42f-6de1-4f5a-8115-50f8b2ff573f\") " pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:10:35.686069 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:35.686022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vvqt\" (UniqueName: \"kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt\") pod \"network-check-target-qnvmg\" (UID: \"b2d2669b-897a-4fdf-8df5-0cd56e7d47a5\") " pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:10:35.688647 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:35.688630    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 03:10:35.698476 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:35.698456    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 03:10:35.709524 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:35.709502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vvqt\" (UniqueName: \"kubernetes.io/projected/b2d2669b-897a-4fdf-8df5-0cd56e7d47a5-kube-api-access-4vvqt\") pod \"network-check-target-qnvmg\" (UID: \"b2d2669b-897a-4fdf-8df5-0cd56e7d47a5\") " pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:10:35.855020 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:35.854941    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcg5l"
Aug 04 03:10:35.883344 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:35.883317    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jb5ws\""
Aug 04 03:10:35.891445 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:35.891413    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:10:36.007291 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:36.007260    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gcg5l"]
Aug 04 03:10:36.010569 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:10:36.010531    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b8b42f_6de1_4f5a_8115_50f8b2ff573f.slice/crio-d5f37179edb93292d19b601e0f38d4d8d22aa71f86190ee19c8e7a7e7abc6cc0 WatchSource:0}: Error finding container d5f37179edb93292d19b601e0f38d4d8d22aa71f86190ee19c8e7a7e7abc6cc0: Status 404 returned error can't find the container with id d5f37179edb93292d19b601e0f38d4d8d22aa71f86190ee19c8e7a7e7abc6cc0
Aug 04 03:10:36.035085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:36.035060    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qnvmg"]
Aug 04 03:10:36.037706 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:10:36.037683    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d2669b_897a_4fdf_8df5_0cd56e7d47a5.slice/crio-a72eec144c0891df5acaaec853ad9370395ff2d4553d6328700895ce5874dccb WatchSource:0}: Error finding container a72eec144c0891df5acaaec853ad9370395ff2d4553d6328700895ce5874dccb: Status 404 returned error can't find the container with id a72eec144c0891df5acaaec853ad9370395ff2d4553d6328700895ce5874dccb
Aug 04 03:10:36.075831 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:36.075801    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qnvmg" event={"ID":"b2d2669b-897a-4fdf-8df5-0cd56e7d47a5","Type":"ContainerStarted","Data":"a72eec144c0891df5acaaec853ad9370395ff2d4553d6328700895ce5874dccb"}
Aug 04 03:10:36.076748 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:36.076727    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gcg5l" event={"ID":"54b8b42f-6de1-4f5a-8115-50f8b2ff573f","Type":"ContainerStarted","Data":"d5f37179edb93292d19b601e0f38d4d8d22aa71f86190ee19c8e7a7e7abc6cc0"}
Aug 04 03:10:36.996296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:36.996240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-2t9qv\" (UID: \"9c78063e-e4f9-446b-90c5-89f0f2ddc001\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"
Aug 04 03:10:36.996736 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:36.996334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:10:36.996736 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:36.996486    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 03:10:36.996736 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:36.996501    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-958dbfdbb-wz2mz: secret "image-registry-tls" not found
Aug 04 03:10:36.996736 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:36.996574    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls podName:5f607efb-3bb4-4755-a366-4d8b0c5aa431 nodeName:}" failed. No retries permitted until 2026-08-04 03:11:08.99655205 +0000 UTC m=+99.676339149 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls") pod "image-registry-958dbfdbb-wz2mz" (UID: "5f607efb-3bb4-4755-a366-4d8b0c5aa431") : secret "image-registry-tls" not found
Aug 04 03:10:36.996975 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:36.996941    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 03:10:36.997040 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:36.997028    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert podName:9c78063e-e4f9-446b-90c5-89f0f2ddc001 nodeName:}" failed. No retries permitted until 2026-08-04 03:11:08.997010596 +0000 UTC m=+99.676797706 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-2t9qv" (UID: "9c78063e-e4f9-446b-90c5-89f0f2ddc001") : secret "networking-console-plugin-cert" not found
Aug 04 03:10:37.097627 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:37.097576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:10:37.097806 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:37.097761    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 03:10:37.097872 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:10:37.097818    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls podName:7b43cf1f-077d-485b-8a0d-fdec861aff43 nodeName:}" failed. No retries permitted until 2026-08-04 03:11:09.097799914 +0000 UTC m=+99.777587032 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls") pod "dns-default-np56j" (UID: "7b43cf1f-077d-485b-8a0d-fdec861aff43") : secret "dns-default-metrics-tls" not found
Aug 04 03:10:41.091605 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:41.091508    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qnvmg" event={"ID":"b2d2669b-897a-4fdf-8df5-0cd56e7d47a5","Type":"ContainerStarted","Data":"7e0aa56d2114bc2d32e5d212f8ba3bd14130fe6c4f2fd854db3c8531b520916e"}
Aug 04 03:10:41.091605 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:41.091570    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:10:41.092881 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:41.092858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gcg5l" event={"ID":"54b8b42f-6de1-4f5a-8115-50f8b2ff573f","Type":"ContainerStarted","Data":"bcf32baf1792583f98acf40e10c46ce10bd604b145a35c7c5eee026e73ec2ae0"}
Aug 04 03:10:41.106063 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:41.106022    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qnvmg" podStartSLOduration=66.390376921 podStartE2EDuration="1m11.106009127s" totalImagesPullingTime="4.715632206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:10:41.105324638 +0000 UTC m=+71.785111757" watchObservedRunningTime="2026-08-04 03:10:41.106009127 +0000 UTC m=+71.785796246"
Aug 04 03:10:41.119438 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:10:41.119400    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gcg5l" podStartSLOduration=66.378163837 podStartE2EDuration="1m11.119389199s" totalImagesPullingTime="4.741225362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:10:41.119145548 +0000 UTC m=+71.798932668" watchObservedRunningTime="2026-08-04 03:10:41.119389199 +0000 UTC m=+71.799176380"
Aug 04 03:11:09.030995 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:11:09.030909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:11:09.031377 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:11:09.031060    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 03:11:09.031377 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:11:09.031079    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-958dbfdbb-wz2mz: secret "image-registry-tls" not found
Aug 04 03:11:09.031377 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:11:09.031092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-2t9qv\" (UID: \"9c78063e-e4f9-446b-90c5-89f0f2ddc001\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"
Aug 04 03:11:09.031377 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:11:09.031145    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls podName:5f607efb-3bb4-4755-a366-4d8b0c5aa431 nodeName:}" failed. No retries permitted until 2026-08-04 03:12:13.03112824 +0000 UTC m=+163.710915337 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls") pod "image-registry-958dbfdbb-wz2mz" (UID: "5f607efb-3bb4-4755-a366-4d8b0c5aa431") : secret "image-registry-tls" not found
Aug 04 03:11:09.031377 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:11:09.031183    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 03:11:09.031377 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:11:09.031236    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert podName:9c78063e-e4f9-446b-90c5-89f0f2ddc001 nodeName:}" failed. No retries permitted until 2026-08-04 03:12:13.031224143 +0000 UTC m=+163.711011240 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-2t9qv" (UID: "9c78063e-e4f9-446b-90c5-89f0f2ddc001") : secret "networking-console-plugin-cert" not found
Aug 04 03:11:09.131572 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:11:09.131535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:11:09.131733 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:11:09.131690    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 03:11:09.131772 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:11:09.131766    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls podName:7b43cf1f-077d-485b-8a0d-fdec861aff43 nodeName:}" failed. No retries permitted until 2026-08-04 03:12:13.131749388 +0000 UTC m=+163.811536484 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls") pod "dns-default-np56j" (UID: "7b43cf1f-077d-485b-8a0d-fdec861aff43") : secret "dns-default-metrics-tls" not found
Aug 04 03:11:12.097000 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:11:12.096956    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qnvmg"
Aug 04 03:11:39.556496 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:11:39.556445    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs\") pod \"network-metrics-daemon-c5d74\" (UID: \"b39560be-b9af-4ff7-8a5b-7882c66a03d8\") " pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:11:39.556922 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:11:39.556608    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Aug 04 03:11:39.556922 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:11:39.556686    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs podName:b39560be-b9af-4ff7-8a5b-7882c66a03d8 nodeName:}" failed. No retries permitted until 2026-08-04 03:13:41.556670418 +0000 UTC m=+252.236457515 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs") pod "network-metrics-daemon-c5d74" (UID: "b39560be-b9af-4ff7-8a5b-7882c66a03d8") : secret "metrics-daemon-secret" not found
Aug 04 03:12:08.181528 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:12:08.181481    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv" podUID="9c78063e-e4f9-446b-90c5-89f0f2ddc001"
Aug 04 03:12:08.192634 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:12:08.192591    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz" podUID="5f607efb-3bb4-4755-a366-4d8b0c5aa431"
Aug 04 03:12:08.205731 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:12:08.205710    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-np56j" podUID="7b43cf1f-077d-485b-8a0d-fdec861aff43"
Aug 04 03:12:08.295188 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:12:08.295099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-np56j"
Aug 04 03:12:08.295310 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:12:08.295099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"
Aug 04 03:12:08.295310 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:12:08.295108    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:12:08.872906 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:12:08.872857    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:12:13.094798 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:12:13.094762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-2t9qv\" (UID: \"9c78063e-e4f9-446b-90c5-89f0f2ddc001\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"
Aug 04 03:12:13.095316 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:12:13.094820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:12:13.095316 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:12:13.094923    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 03:12:13.095316 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:12:13.094929    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 03:12:13.095316 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:12:13.094989    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-958dbfdbb-wz2mz: secret "image-registry-tls" not found
Aug 04 03:12:13.095316 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:12:13.095012    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert podName:9c78063e-e4f9-446b-90c5-89f0f2ddc001 nodeName:}" failed. No retries permitted until 2026-08-04 03:14:15.094993319 +0000 UTC m=+285.774780420 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-2t9qv" (UID: "9c78063e-e4f9-446b-90c5-89f0f2ddc001") : secret "networking-console-plugin-cert" not found
Aug 04 03:12:13.095316 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:12:13.095051    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls podName:5f607efb-3bb4-4755-a366-4d8b0c5aa431 nodeName:}" failed. No retries permitted until 2026-08-04 03:14:15.095033671 +0000 UTC m=+285.774820770 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls") pod "image-registry-958dbfdbb-wz2mz" (UID: "5f607efb-3bb4-4755-a366-4d8b0c5aa431") : secret "image-registry-tls" not found
Aug 04 03:12:13.196188 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:12:13.196150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:12:13.196349 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:12:13.196259    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 03:12:13.196349 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:12:13.196308    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls podName:7b43cf1f-077d-485b-8a0d-fdec861aff43 nodeName:}" failed. No retries permitted until 2026-08-04 03:14:15.196294505 +0000 UTC m=+285.876081603 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls") pod "dns-default-np56j" (UID: "7b43cf1f-077d-485b-8a0d-fdec861aff43") : secret "dns-default-metrics-tls" not found
Aug 04 03:12:20.839252 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:12:20.839210    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:13:41.612117 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:41.612055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs\") pod \"network-metrics-daemon-c5d74\" (UID: \"b39560be-b9af-4ff7-8a5b-7882c66a03d8\") " pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:13:41.612621 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:41.612213    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Aug 04 03:13:41.612621 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:41.612278    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs podName:b39560be-b9af-4ff7-8a5b-7882c66a03d8 nodeName:}" failed. No retries permitted until 2026-08-04 03:15:43.612263308 +0000 UTC m=+374.292050404 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs") pod "network-metrics-daemon-c5d74" (UID: "b39560be-b9af-4ff7-8a5b-7882c66a03d8") : secret "metrics-daemon-secret" not found
Aug 04 03:13:51.161296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.161260    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-6kmw8"]
Aug 04 03:13:51.162934 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.162917    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb"]
Aug 04 03:13:51.163080 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.163061    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6kmw8"
Aug 04 03:13:51.164858 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.164841    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb"
Aug 04 03:13:51.165156 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.165132    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2k8hf"]
Aug 04 03:13:51.165451 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.165431    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Aug 04 03:13:51.165528 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.165478    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Aug 04 03:13:51.165528 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.165521    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Aug 04 03:13:51.165635 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.165521    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-lfknj\""
Aug 04 03:13:51.165828 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.165815    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Aug 04 03:13:51.166820 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.166805    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk"]
Aug 04 03:13:51.166943 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.166925    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2k8hf"
Aug 04 03:13:51.168487 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.168465    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Aug 04 03:13:51.168487 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.168483    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Aug 04 03:13:51.168652 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.168504    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Aug 04 03:13:51.168652 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.168471    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-schv8\""
Aug 04 03:13:51.168758 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.168721    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk"
Aug 04 03:13:51.168807 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.168508    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Aug 04 03:13:51.173854 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.173839    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Aug 04 03:13:51.174123 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.174108    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xj5gp\""
Aug 04 03:13:51.174192 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.174136    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gcrgs\""
Aug 04 03:13:51.174369 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.174352    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Aug 04 03:13:51.174457 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.174382    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Aug 04 03:13:51.175620 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.175605    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Aug 04 03:13:51.176346 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.176325    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-6kmw8"]
Aug 04 03:13:51.176944 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.176929    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Aug 04 03:13:51.178696 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.178679    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb"]
Aug 04 03:13:51.181645 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.181623    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2k8hf"]
Aug 04 03:13:51.198609 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.198587    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk"]
Aug 04 03:13:51.263587 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.263516    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-2v2qn"]
Aug 04 03:13:51.265541 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.265524    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2v2qn"
Aug 04 03:13:51.267883 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.267863    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6s8hc\""
Aug 04 03:13:51.273765 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.273745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpqr7\" (UniqueName: \"kubernetes.io/projected/6b243e88-7cd6-4eae-ad2e-399c8b43d676-kube-api-access-tpqr7\") pod \"service-ca-operator-784f97bcf6-6kmw8\" (UID: \"6b243e88-7cd6-4eae-ad2e-399c8b43d676\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6kmw8"
Aug 04 03:13:51.273855 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.273772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b243e88-7cd6-4eae-ad2e-399c8b43d676-serving-cert\") pod \"service-ca-operator-784f97bcf6-6kmw8\" (UID: \"6b243e88-7cd6-4eae-ad2e-399c8b43d676\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6kmw8"
Aug 04 03:13:51.273855 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.273805    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9q29\" (UniqueName: \"kubernetes.io/projected/c3e1258f-b644-4469-b525-eccc09288622-kube-api-access-c9q29\") pod \"volume-data-source-validator-5b8776669-2k8hf\" (UID: \"c3e1258f-b644-4469-b525-eccc09288622\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2k8hf"
Aug 04 03:13:51.273855 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.273822    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-qgpdb\" (UID: \"f2d6c41f-c453-40d7-be08-ea72a2fcf37d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb"
Aug 04 03:13:51.273995 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.273867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-qgpdb\" (UID: \"f2d6c41f-c453-40d7-be08-ea72a2fcf37d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb"
Aug 04 03:13:51.273995 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.273899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-nj2gk\" (UID: \"2b1f4754-8f2a-4da5-96db-db8a6b6b88fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk"
Aug 04 03:13:51.273995 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.273913    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b243e88-7cd6-4eae-ad2e-399c8b43d676-config\") pod \"service-ca-operator-784f97bcf6-6kmw8\" (UID: \"6b243e88-7cd6-4eae-ad2e-399c8b43d676\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6kmw8"
Aug 04 03:13:51.273995 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.273935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv84w\" (UniqueName: \"kubernetes.io/projected/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-kube-api-access-pv84w\") pod \"cluster-samples-operator-559cbf5745-nj2gk\" (UID: \"2b1f4754-8f2a-4da5-96db-db8a6b6b88fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk"
Aug 04 03:13:51.274132 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.274080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26ppv\" (UniqueName: \"kubernetes.io/projected/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-kube-api-access-26ppv\") pod \"cluster-monitoring-operator-6f55c4f58d-qgpdb\" (UID: \"f2d6c41f-c453-40d7-be08-ea72a2fcf37d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb"
Aug 04 03:13:51.279801 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.279780    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-2v2qn"]
Aug 04 03:13:51.375346 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.375311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26ppv\" (UniqueName: \"kubernetes.io/projected/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-kube-api-access-26ppv\") pod \"cluster-monitoring-operator-6f55c4f58d-qgpdb\" (UID: \"f2d6c41f-c453-40d7-be08-ea72a2fcf37d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb"
Aug 04 03:13:51.375346 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.375347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpqr7\" (UniqueName: \"kubernetes.io/projected/6b243e88-7cd6-4eae-ad2e-399c8b43d676-kube-api-access-tpqr7\") pod \"service-ca-operator-784f97bcf6-6kmw8\" (UID: \"6b243e88-7cd6-4eae-ad2e-399c8b43d676\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6kmw8"
Aug 04 03:13:51.375579 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.375365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b243e88-7cd6-4eae-ad2e-399c8b43d676-serving-cert\") pod \"service-ca-operator-784f97bcf6-6kmw8\" (UID: \"6b243e88-7cd6-4eae-ad2e-399c8b43d676\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6kmw8"
Aug 04 03:13:51.375579 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.375499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9q29\" (UniqueName: \"kubernetes.io/projected/c3e1258f-b644-4469-b525-eccc09288622-kube-api-access-c9q29\") pod \"volume-data-source-validator-5b8776669-2k8hf\" (UID: \"c3e1258f-b644-4469-b525-eccc09288622\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2k8hf"
Aug 04 03:13:51.375579 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.375531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-qgpdb\" (UID: \"f2d6c41f-c453-40d7-be08-ea72a2fcf37d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb"
Aug 04 03:13:51.375579 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.375555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-qgpdb\" (UID: \"f2d6c41f-c453-40d7-be08-ea72a2fcf37d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb"
Aug 04 03:13:51.375763 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.375584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-nj2gk\" (UID: \"2b1f4754-8f2a-4da5-96db-db8a6b6b88fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk"
Aug 04 03:13:51.375763 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.375620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b243e88-7cd6-4eae-ad2e-399c8b43d676-config\") pod \"service-ca-operator-784f97bcf6-6kmw8\" (UID: \"6b243e88-7cd6-4eae-ad2e-399c8b43d676\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6kmw8"
Aug 04 03:13:51.375763 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.375657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv84w\" (UniqueName: \"kubernetes.io/projected/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-kube-api-access-pv84w\") pod \"cluster-samples-operator-559cbf5745-nj2gk\" (UID: \"2b1f4754-8f2a-4da5-96db-db8a6b6b88fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk"
Aug 04 03:13:51.375763 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:51.375669    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Aug 04 03:13:51.375763 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.375690    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlnvj\" (UniqueName: \"kubernetes.io/projected/baa57911-7fda-4e4e-a53d-74686c0a6159-kube-api-access-wlnvj\") pod \"network-check-source-65bdbbff45-2v2qn\" (UID: \"baa57911-7fda-4e4e-a53d-74686c0a6159\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2v2qn"
Aug 04 03:13:51.375763 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:51.375737    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-samples-operator-tls podName:2b1f4754-8f2a-4da5-96db-db8a6b6b88fe nodeName:}" failed. No retries permitted until 2026-08-04 03:13:51.875719572 +0000 UTC m=+262.555506669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-nj2gk" (UID: "2b1f4754-8f2a-4da5-96db-db8a6b6b88fe") : secret "samples-operator-tls" not found
Aug 04 03:13:51.375763 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:51.375672    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 03:13:51.376164 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:51.375811    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-cluster-monitoring-operator-tls podName:f2d6c41f-c453-40d7-be08-ea72a2fcf37d nodeName:}" failed. No retries permitted until 2026-08-04 03:13:51.875793441 +0000 UTC m=+262.555580540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-qgpdb" (UID: "f2d6c41f-c453-40d7-be08-ea72a2fcf37d") : secret "cluster-monitoring-operator-tls" not found
Aug 04 03:13:51.376164 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.376132    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b243e88-7cd6-4eae-ad2e-399c8b43d676-config\") pod \"service-ca-operator-784f97bcf6-6kmw8\" (UID: \"6b243e88-7cd6-4eae-ad2e-399c8b43d676\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6kmw8"
Aug 04 03:13:51.376340 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.376321    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-qgpdb\" (UID: \"f2d6c41f-c453-40d7-be08-ea72a2fcf37d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb"
Aug 04 03:13:51.377978 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.377948    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b243e88-7cd6-4eae-ad2e-399c8b43d676-serving-cert\") pod \"service-ca-operator-784f97bcf6-6kmw8\" (UID: \"6b243e88-7cd6-4eae-ad2e-399c8b43d676\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6kmw8"
Aug 04 03:13:51.384242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.384217    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26ppv\" (UniqueName: \"kubernetes.io/projected/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-kube-api-access-26ppv\") pod \"cluster-monitoring-operator-6f55c4f58d-qgpdb\" (UID: \"f2d6c41f-c453-40d7-be08-ea72a2fcf37d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb"
Aug 04 03:13:51.384458 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.384432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9q29\" (UniqueName: \"kubernetes.io/projected/c3e1258f-b644-4469-b525-eccc09288622-kube-api-access-c9q29\") pod \"volume-data-source-validator-5b8776669-2k8hf\" (UID: \"c3e1258f-b644-4469-b525-eccc09288622\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2k8hf"
Aug 04 03:13:51.384520 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.384466    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv84w\" (UniqueName: \"kubernetes.io/projected/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-kube-api-access-pv84w\") pod \"cluster-samples-operator-559cbf5745-nj2gk\" (UID: \"2b1f4754-8f2a-4da5-96db-db8a6b6b88fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk"
Aug 04 03:13:51.384520 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.384506    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpqr7\" (UniqueName: \"kubernetes.io/projected/6b243e88-7cd6-4eae-ad2e-399c8b43d676-kube-api-access-tpqr7\") pod \"service-ca-operator-784f97bcf6-6kmw8\" (UID: \"6b243e88-7cd6-4eae-ad2e-399c8b43d676\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6kmw8"
Aug 04 03:13:51.473310 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.473279    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6kmw8"
Aug 04 03:13:51.477085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.477064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlnvj\" (UniqueName: \"kubernetes.io/projected/baa57911-7fda-4e4e-a53d-74686c0a6159-kube-api-access-wlnvj\") pod \"network-check-source-65bdbbff45-2v2qn\" (UID: \"baa57911-7fda-4e4e-a53d-74686c0a6159\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2v2qn"
Aug 04 03:13:51.485427 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.485400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlnvj\" (UniqueName: \"kubernetes.io/projected/baa57911-7fda-4e4e-a53d-74686c0a6159-kube-api-access-wlnvj\") pod \"network-check-source-65bdbbff45-2v2qn\" (UID: \"baa57911-7fda-4e4e-a53d-74686c0a6159\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2v2qn"
Aug 04 03:13:51.488047 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.488023    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2k8hf"
Aug 04 03:13:51.573308 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.573281    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2v2qn"
Aug 04 03:13:51.604527 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.604504    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-6kmw8"]
Aug 04 03:13:51.607586 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:13:51.607550    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b243e88_7cd6_4eae_ad2e_399c8b43d676.slice/crio-7f11fbf2985e54724a41045523192c5b8c67711d592776bf37be19ab30603712 WatchSource:0}: Error finding container 7f11fbf2985e54724a41045523192c5b8c67711d592776bf37be19ab30603712: Status 404 returned error can't find the container with id 7f11fbf2985e54724a41045523192c5b8c67711d592776bf37be19ab30603712
Aug 04 03:13:51.620524 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.620489    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2k8hf"]
Aug 04 03:13:51.624078 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:13:51.624053    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e1258f_b644_4469_b525_eccc09288622.slice/crio-4e7151acbd83961e0e58a00ddbd4ca50d135ed78eba3ecf1707118ccae056c49 WatchSource:0}: Error finding container 4e7151acbd83961e0e58a00ddbd4ca50d135ed78eba3ecf1707118ccae056c49: Status 404 returned error can't find the container with id 4e7151acbd83961e0e58a00ddbd4ca50d135ed78eba3ecf1707118ccae056c49
Aug 04 03:13:51.702464 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.702434    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-2v2qn"]
Aug 04 03:13:51.705554 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:13:51.705528    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa57911_7fda_4e4e_a53d_74686c0a6159.slice/crio-03fd4997eb99b345a829d1349131172ec61b7f5edda8f49e877da40f137e17c5 WatchSource:0}: Error finding container 03fd4997eb99b345a829d1349131172ec61b7f5edda8f49e877da40f137e17c5: Status 404 returned error can't find the container with id 03fd4997eb99b345a829d1349131172ec61b7f5edda8f49e877da40f137e17c5
Aug 04 03:13:51.880724 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.880693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-qgpdb\" (UID: \"f2d6c41f-c453-40d7-be08-ea72a2fcf37d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb"
Aug 04 03:13:51.880724 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:51.880727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-nj2gk\" (UID: \"2b1f4754-8f2a-4da5-96db-db8a6b6b88fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk"
Aug 04 03:13:51.880956 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:51.880841    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 03:13:51.880956 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:51.880882    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Aug 04 03:13:51.880956 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:51.880907    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-cluster-monitoring-operator-tls podName:f2d6c41f-c453-40d7-be08-ea72a2fcf37d nodeName:}" failed. No retries permitted until 2026-08-04 03:13:52.880888654 +0000 UTC m=+263.560675756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-qgpdb" (UID: "f2d6c41f-c453-40d7-be08-ea72a2fcf37d") : secret "cluster-monitoring-operator-tls" not found
Aug 04 03:13:51.880956 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:51.880935    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-samples-operator-tls podName:2b1f4754-8f2a-4da5-96db-db8a6b6b88fe nodeName:}" failed. No retries permitted until 2026-08-04 03:13:52.880918344 +0000 UTC m=+263.560705450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-nj2gk" (UID: "2b1f4754-8f2a-4da5-96db-db8a6b6b88fe") : secret "samples-operator-tls" not found
Aug 04 03:13:52.534886 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:52.534796    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2v2qn" event={"ID":"baa57911-7fda-4e4e-a53d-74686c0a6159","Type":"ContainerStarted","Data":"f51a29f4f71fbf1b4e887ade3985d8f42b8346a8987b734f36adf559f470b150"}
Aug 04 03:13:52.534886 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:52.534860    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2v2qn" event={"ID":"baa57911-7fda-4e4e-a53d-74686c0a6159","Type":"ContainerStarted","Data":"03fd4997eb99b345a829d1349131172ec61b7f5edda8f49e877da40f137e17c5"}
Aug 04 03:13:52.536630 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:52.536577    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6kmw8" event={"ID":"6b243e88-7cd6-4eae-ad2e-399c8b43d676","Type":"ContainerStarted","Data":"7f11fbf2985e54724a41045523192c5b8c67711d592776bf37be19ab30603712"}
Aug 04 03:13:52.538219 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:52.538191    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2k8hf" event={"ID":"c3e1258f-b644-4469-b525-eccc09288622","Type":"ContainerStarted","Data":"4e7151acbd83961e0e58a00ddbd4ca50d135ed78eba3ecf1707118ccae056c49"}
Aug 04 03:13:52.889143 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:52.889108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-qgpdb\" (UID: \"f2d6c41f-c453-40d7-be08-ea72a2fcf37d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb"
Aug 04 03:13:52.889322 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:52.889158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-nj2gk\" (UID: \"2b1f4754-8f2a-4da5-96db-db8a6b6b88fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk"
Aug 04 03:13:52.889322 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:52.889269    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Aug 04 03:13:52.889322 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:52.889272    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 03:13:52.889498 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:52.889333    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-samples-operator-tls podName:2b1f4754-8f2a-4da5-96db-db8a6b6b88fe nodeName:}" failed. No retries permitted until 2026-08-04 03:13:54.889319929 +0000 UTC m=+265.569107026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-nj2gk" (UID: "2b1f4754-8f2a-4da5-96db-db8a6b6b88fe") : secret "samples-operator-tls" not found
Aug 04 03:13:52.889498 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:52.889348    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-cluster-monitoring-operator-tls podName:f2d6c41f-c453-40d7-be08-ea72a2fcf37d nodeName:}" failed. No retries permitted until 2026-08-04 03:13:54.889341295 +0000 UTC m=+265.569128391 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-qgpdb" (UID: "f2d6c41f-c453-40d7-be08-ea72a2fcf37d") : secret "cluster-monitoring-operator-tls" not found
Aug 04 03:13:53.542921 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:53.542876    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2k8hf" event={"ID":"c3e1258f-b644-4469-b525-eccc09288622","Type":"ContainerStarted","Data":"1889b91b559026ab0926b55b866991780ba95e186afe5cfd38966d29872903f8"}
Aug 04 03:13:53.558866 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:53.558806    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2k8hf" podStartSLOduration=1.462195122 podStartE2EDuration="2.558790797s" totalImagesPullingTime="1.096595675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:13:53.557199372 +0000 UTC m=+264.236986517" watchObservedRunningTime="2026-08-04 03:13:53.558790797 +0000 UTC m=+264.238577915"
Aug 04 03:13:53.559053 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:53.558895    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2v2qn" podStartSLOduration=2.55888919 podStartE2EDuration="2.55888919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:13:52.55032966 +0000 UTC m=+263.230116786" watchObservedRunningTime="2026-08-04 03:13:53.55888919 +0000 UTC m=+264.238676308"
Aug 04 03:13:54.547343 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:54.547304    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6kmw8" event={"ID":"6b243e88-7cd6-4eae-ad2e-399c8b43d676","Type":"ContainerStarted","Data":"ba3eecd4748cdaef184435139e69ef2cc77345e6cc091cd5bb2e674cfd7affbc"}
Aug 04 03:13:54.562662 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:54.562616    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6kmw8" podStartSLOduration=1.470652459 podStartE2EDuration="3.562601688s" totalImagesPullingTime="2.091949229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:13:54.562056304 +0000 UTC m=+265.241843439" watchObservedRunningTime="2026-08-04 03:13:54.562601688 +0000 UTC m=+265.242388807"
Aug 04 03:13:54.904387 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:54.904304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-qgpdb\" (UID: \"f2d6c41f-c453-40d7-be08-ea72a2fcf37d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb"
Aug 04 03:13:54.904387 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:54.904342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-nj2gk\" (UID: \"2b1f4754-8f2a-4da5-96db-db8a6b6b88fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk"
Aug 04 03:13:54.904572 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:54.904460    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 03:13:54.904572 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:54.904527    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-cluster-monitoring-operator-tls podName:f2d6c41f-c453-40d7-be08-ea72a2fcf37d nodeName:}" failed. No retries permitted until 2026-08-04 03:13:58.904511814 +0000 UTC m=+269.584298911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-qgpdb" (UID: "f2d6c41f-c453-40d7-be08-ea72a2fcf37d") : secret "cluster-monitoring-operator-tls" not found
Aug 04 03:13:54.904572 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:54.904468    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Aug 04 03:13:54.904675 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:54.904596    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-samples-operator-tls podName:2b1f4754-8f2a-4da5-96db-db8a6b6b88fe nodeName:}" failed. No retries permitted until 2026-08-04 03:13:58.904582284 +0000 UTC m=+269.584369381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-nj2gk" (UID: "2b1f4754-8f2a-4da5-96db-db8a6b6b88fe") : secret "samples-operator-tls" not found
Aug 04 03:13:55.963284 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:55.963247    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rzf5k"]
Aug 04 03:13:55.965485 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:55.965454    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:55.967826 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:55.967802    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Aug 04 03:13:55.967826 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:55.967819    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Aug 04 03:13:55.968041 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:55.967873    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Aug 04 03:13:55.968174 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:55.968155    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Aug 04 03:13:55.969144 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:55.969129    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2m679\""
Aug 04 03:13:55.976658 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:55.976640    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rzf5k"]
Aug 04 03:13:56.012310 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:56.012278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5xhj\" (UniqueName: \"kubernetes.io/projected/220034f7-6232-4501-8842-f79fd919fc3c-kube-api-access-j5xhj\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:56.012467 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:56.012318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/220034f7-6232-4501-8842-f79fd919fc3c-crio-socket\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:56.012467 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:56.012366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/220034f7-6232-4501-8842-f79fd919fc3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:56.012467 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:56.012415    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/220034f7-6232-4501-8842-f79fd919fc3c-data-volume\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:56.012467 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:56.012440    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/220034f7-6232-4501-8842-f79fd919fc3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:56.113237 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:56.113207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5xhj\" (UniqueName: \"kubernetes.io/projected/220034f7-6232-4501-8842-f79fd919fc3c-kube-api-access-j5xhj\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:56.113430 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:56.113251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/220034f7-6232-4501-8842-f79fd919fc3c-crio-socket\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:56.113430 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:56.113339    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/220034f7-6232-4501-8842-f79fd919fc3c-crio-socket\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:56.113430 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:56.113394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/220034f7-6232-4501-8842-f79fd919fc3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:56.113589 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:56.113447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/220034f7-6232-4501-8842-f79fd919fc3c-data-volume\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:56.113589 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:56.113473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/220034f7-6232-4501-8842-f79fd919fc3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:56.113589 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:56.113509    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 03:13:56.113589 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:56.113556    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220034f7-6232-4501-8842-f79fd919fc3c-insights-runtime-extractor-tls podName:220034f7-6232-4501-8842-f79fd919fc3c nodeName:}" failed. No retries permitted until 2026-08-04 03:13:56.613538234 +0000 UTC m=+267.293325331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/220034f7-6232-4501-8842-f79fd919fc3c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rzf5k" (UID: "220034f7-6232-4501-8842-f79fd919fc3c") : secret "insights-runtime-extractor-tls" not found
Aug 04 03:13:56.113589 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:56.113565    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 03:13:56.113835 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:56.113619    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/220034f7-6232-4501-8842-f79fd919fc3c-kube-rbac-proxy-cm podName:220034f7-6232-4501-8842-f79fd919fc3c nodeName:}" failed. No retries permitted until 2026-08-04 03:13:56.613603054 +0000 UTC m=+267.293390151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/220034f7-6232-4501-8842-f79fd919fc3c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rzf5k" (UID: "220034f7-6232-4501-8842-f79fd919fc3c") : configmap "kube-rbac-proxy" not found
Aug 04 03:13:56.113835 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:56.113747    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/220034f7-6232-4501-8842-f79fd919fc3c-data-volume\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:56.121201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:56.121175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5xhj\" (UniqueName: \"kubernetes.io/projected/220034f7-6232-4501-8842-f79fd919fc3c-kube-api-access-j5xhj\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:56.618136 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:56.618104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/220034f7-6232-4501-8842-f79fd919fc3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:56.618372 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:56.618150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/220034f7-6232-4501-8842-f79fd919fc3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:56.618372 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:56.618257    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 03:13:56.618372 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:56.618294    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 03:13:56.618372 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:56.618329    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220034f7-6232-4501-8842-f79fd919fc3c-insights-runtime-extractor-tls podName:220034f7-6232-4501-8842-f79fd919fc3c nodeName:}" failed. No retries permitted until 2026-08-04 03:13:57.61830742 +0000 UTC m=+268.298094517 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/220034f7-6232-4501-8842-f79fd919fc3c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rzf5k" (UID: "220034f7-6232-4501-8842-f79fd919fc3c") : secret "insights-runtime-extractor-tls" not found
Aug 04 03:13:56.618372 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:56.618347    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/220034f7-6232-4501-8842-f79fd919fc3c-kube-rbac-proxy-cm podName:220034f7-6232-4501-8842-f79fd919fc3c nodeName:}" failed. No retries permitted until 2026-08-04 03:13:57.618335647 +0000 UTC m=+268.298122749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/220034f7-6232-4501-8842-f79fd919fc3c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rzf5k" (UID: "220034f7-6232-4501-8842-f79fd919fc3c") : configmap "kube-rbac-proxy" not found
Aug 04 03:13:57.140976 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.140927    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-7m7fl"]
Aug 04 03:13:57.144176 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.144157    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-7m7fl"
Aug 04 03:13:57.146827 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.146805    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Aug 04 03:13:57.146911 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.146833    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Aug 04 03:13:57.147863 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.147849    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mf9pt\""
Aug 04 03:13:57.147910 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.147894    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Aug 04 03:13:57.147952 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.147928    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Aug 04 03:13:57.153362 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.153342    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-7m7fl"]
Aug 04 03:13:57.223823 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.223784    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/48b24d73-7d9f-4890-b5d1-c282a34c58e1-signing-key\") pod \"service-ca-98c88fbb5-7m7fl\" (UID: \"48b24d73-7d9f-4890-b5d1-c282a34c58e1\") " pod="openshift-service-ca/service-ca-98c88fbb5-7m7fl"
Aug 04 03:13:57.223823 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.223820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvttw\" (UniqueName: \"kubernetes.io/projected/48b24d73-7d9f-4890-b5d1-c282a34c58e1-kube-api-access-lvttw\") pod \"service-ca-98c88fbb5-7m7fl\" (UID: \"48b24d73-7d9f-4890-b5d1-c282a34c58e1\") " pod="openshift-service-ca/service-ca-98c88fbb5-7m7fl"
Aug 04 03:13:57.224052 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.223893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/48b24d73-7d9f-4890-b5d1-c282a34c58e1-signing-cabundle\") pod \"service-ca-98c88fbb5-7m7fl\" (UID: \"48b24d73-7d9f-4890-b5d1-c282a34c58e1\") " pod="openshift-service-ca/service-ca-98c88fbb5-7m7fl"
Aug 04 03:13:57.242037 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.242006    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-gzkwr"]
Aug 04 03:13:57.245020 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.245005    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-gzkwr"
Aug 04 03:13:57.247622 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.247601    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4zzvx\""
Aug 04 03:13:57.247732 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.247686    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Aug 04 03:13:57.247732 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.247721    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Aug 04 03:13:57.252646 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.252627    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-gzkwr"]
Aug 04 03:13:57.324864 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.324825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glbrs\" (UniqueName: \"kubernetes.io/projected/3c397a61-27ed-4be0-b608-2ab09c04b5a5-kube-api-access-glbrs\") pod \"migrator-68bc745d87-gzkwr\" (UID: \"3c397a61-27ed-4be0-b608-2ab09c04b5a5\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-gzkwr"
Aug 04 03:13:57.325072 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.324893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/48b24d73-7d9f-4890-b5d1-c282a34c58e1-signing-key\") pod \"service-ca-98c88fbb5-7m7fl\" (UID: \"48b24d73-7d9f-4890-b5d1-c282a34c58e1\") " pod="openshift-service-ca/service-ca-98c88fbb5-7m7fl"
Aug 04 03:13:57.325072 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.324912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvttw\" (UniqueName: \"kubernetes.io/projected/48b24d73-7d9f-4890-b5d1-c282a34c58e1-kube-api-access-lvttw\") pod \"service-ca-98c88fbb5-7m7fl\" (UID: \"48b24d73-7d9f-4890-b5d1-c282a34c58e1\") " pod="openshift-service-ca/service-ca-98c88fbb5-7m7fl"
Aug 04 03:13:57.325072 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.324948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/48b24d73-7d9f-4890-b5d1-c282a34c58e1-signing-cabundle\") pod \"service-ca-98c88fbb5-7m7fl\" (UID: \"48b24d73-7d9f-4890-b5d1-c282a34c58e1\") " pod="openshift-service-ca/service-ca-98c88fbb5-7m7fl"
Aug 04 03:13:57.325571 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.325545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/48b24d73-7d9f-4890-b5d1-c282a34c58e1-signing-cabundle\") pod \"service-ca-98c88fbb5-7m7fl\" (UID: \"48b24d73-7d9f-4890-b5d1-c282a34c58e1\") " pod="openshift-service-ca/service-ca-98c88fbb5-7m7fl"
Aug 04 03:13:57.327227 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.327206    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/48b24d73-7d9f-4890-b5d1-c282a34c58e1-signing-key\") pod \"service-ca-98c88fbb5-7m7fl\" (UID: \"48b24d73-7d9f-4890-b5d1-c282a34c58e1\") " pod="openshift-service-ca/service-ca-98c88fbb5-7m7fl"
Aug 04 03:13:57.332817 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.332794    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvttw\" (UniqueName: \"kubernetes.io/projected/48b24d73-7d9f-4890-b5d1-c282a34c58e1-kube-api-access-lvttw\") pod \"service-ca-98c88fbb5-7m7fl\" (UID: \"48b24d73-7d9f-4890-b5d1-c282a34c58e1\") " pod="openshift-service-ca/service-ca-98c88fbb5-7m7fl"
Aug 04 03:13:57.425744 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.425643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glbrs\" (UniqueName: \"kubernetes.io/projected/3c397a61-27ed-4be0-b608-2ab09c04b5a5-kube-api-access-glbrs\") pod \"migrator-68bc745d87-gzkwr\" (UID: \"3c397a61-27ed-4be0-b608-2ab09c04b5a5\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-gzkwr"
Aug 04 03:13:57.433930 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.433905    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glbrs\" (UniqueName: \"kubernetes.io/projected/3c397a61-27ed-4be0-b608-2ab09c04b5a5-kube-api-access-glbrs\") pod \"migrator-68bc745d87-gzkwr\" (UID: \"3c397a61-27ed-4be0-b608-2ab09c04b5a5\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-gzkwr"
Aug 04 03:13:57.453006 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.452976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-7m7fl"
Aug 04 03:13:57.554239 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.554213    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-gzkwr"
Aug 04 03:13:57.575000 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.574936    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-7m7fl"]
Aug 04 03:13:57.577395 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:13:57.577360    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b24d73_7d9f_4890_b5d1_c282a34c58e1.slice/crio-aefb3ea1feb5d558c9dc8a8d6c5bf1cbe3043827f326f38ad38ce594be798483 WatchSource:0}: Error finding container aefb3ea1feb5d558c9dc8a8d6c5bf1cbe3043827f326f38ad38ce594be798483: Status 404 returned error can't find the container with id aefb3ea1feb5d558c9dc8a8d6c5bf1cbe3043827f326f38ad38ce594be798483
Aug 04 03:13:57.628099 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.627975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/220034f7-6232-4501-8842-f79fd919fc3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:57.628099 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.628030    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/220034f7-6232-4501-8842-f79fd919fc3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:57.628210 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:57.628106    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 03:13:57.628210 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:57.628134    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 03:13:57.628210 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:57.628173    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/220034f7-6232-4501-8842-f79fd919fc3c-kube-rbac-proxy-cm podName:220034f7-6232-4501-8842-f79fd919fc3c nodeName:}" failed. No retries permitted until 2026-08-04 03:13:59.628153658 +0000 UTC m=+270.307940758 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/220034f7-6232-4501-8842-f79fd919fc3c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rzf5k" (UID: "220034f7-6232-4501-8842-f79fd919fc3c") : configmap "kube-rbac-proxy" not found
Aug 04 03:13:57.628210 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:57.628188    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220034f7-6232-4501-8842-f79fd919fc3c-insights-runtime-extractor-tls podName:220034f7-6232-4501-8842-f79fd919fc3c nodeName:}" failed. No retries permitted until 2026-08-04 03:13:59.628181607 +0000 UTC m=+270.307968704 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/220034f7-6232-4501-8842-f79fd919fc3c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rzf5k" (UID: "220034f7-6232-4501-8842-f79fd919fc3c") : secret "insights-runtime-extractor-tls" not found
Aug 04 03:13:57.687329 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.687256    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-gzkwr"]
Aug 04 03:13:57.690450 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:13:57.690423    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c397a61_27ed_4be0_b608_2ab09c04b5a5.slice/crio-17afe233ff938006ba1ad2fa75c5302180e9a545a2c3ac2bbcb9d1eb96da06e6 WatchSource:0}: Error finding container 17afe233ff938006ba1ad2fa75c5302180e9a545a2c3ac2bbcb9d1eb96da06e6: Status 404 returned error can't find the container with id 17afe233ff938006ba1ad2fa75c5302180e9a545a2c3ac2bbcb9d1eb96da06e6
Aug 04 03:13:57.743256 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:57.743231    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-57qsq_d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8/dns-node-resolver/0.log"
Aug 04 03:13:58.566746 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:58.566708    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-gzkwr" event={"ID":"3c397a61-27ed-4be0-b608-2ab09c04b5a5","Type":"ContainerStarted","Data":"17afe233ff938006ba1ad2fa75c5302180e9a545a2c3ac2bbcb9d1eb96da06e6"}
Aug 04 03:13:58.570609 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:58.570578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-7m7fl" event={"ID":"48b24d73-7d9f-4890-b5d1-c282a34c58e1","Type":"ContainerStarted","Data":"7b5babfc19b6817a017ede3d64ee44976caa7620f026da3cb409f547ed04da1e"}
Aug 04 03:13:58.570739 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:58.570627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-7m7fl" event={"ID":"48b24d73-7d9f-4890-b5d1-c282a34c58e1","Type":"ContainerStarted","Data":"aefb3ea1feb5d558c9dc8a8d6c5bf1cbe3043827f326f38ad38ce594be798483"}
Aug 04 03:13:58.586390 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:58.586345    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-7m7fl" podStartSLOduration=1.586325892 podStartE2EDuration="1.586325892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:13:58.585388664 +0000 UTC m=+269.265175806" watchObservedRunningTime="2026-08-04 03:13:58.586325892 +0000 UTC m=+269.266113012"
Aug 04 03:13:58.940435 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:58.940339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-nj2gk\" (UID: \"2b1f4754-8f2a-4da5-96db-db8a6b6b88fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk"
Aug 04 03:13:58.940435 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:58.940432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-qgpdb\" (UID: \"f2d6c41f-c453-40d7-be08-ea72a2fcf37d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb"
Aug 04 03:13:58.940654 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:58.940504    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Aug 04 03:13:58.940654 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:58.940529    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 03:13:58.940654 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:58.940589    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-samples-operator-tls podName:2b1f4754-8f2a-4da5-96db-db8a6b6b88fe nodeName:}" failed. No retries permitted until 2026-08-04 03:14:06.940566893 +0000 UTC m=+277.620353992 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-nj2gk" (UID: "2b1f4754-8f2a-4da5-96db-db8a6b6b88fe") : secret "samples-operator-tls" not found
Aug 04 03:13:58.940654 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:58.940609    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-cluster-monitoring-operator-tls podName:f2d6c41f-c453-40d7-be08-ea72a2fcf37d nodeName:}" failed. No retries permitted until 2026-08-04 03:14:06.94059905 +0000 UTC m=+277.620386149 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-qgpdb" (UID: "f2d6c41f-c453-40d7-be08-ea72a2fcf37d") : secret "cluster-monitoring-operator-tls" not found
Aug 04 03:13:59.143428 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:59.143400    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gnp8m_5f76ca57-4490-4583-85a2-95198fb43ffd/node-ca/0.log"
Aug 04 03:13:59.576352 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:59.576309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-gzkwr" event={"ID":"3c397a61-27ed-4be0-b608-2ab09c04b5a5","Type":"ContainerStarted","Data":"382b3c6918559cc5b736e78c4d636b2b2448e1ada38984f9299448b068f34188"}
Aug 04 03:13:59.576825 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:59.576378    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-gzkwr" event={"ID":"3c397a61-27ed-4be0-b608-2ab09c04b5a5","Type":"ContainerStarted","Data":"24cc238aed45df1b6095b6d34d6e7c2780f5b2c4105f064ace8a5ee12a7e24f0"}
Aug 04 03:13:59.646692 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:59.646652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/220034f7-6232-4501-8842-f79fd919fc3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:59.646889 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:59.646758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/220034f7-6232-4501-8842-f79fd919fc3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:13:59.647222 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:59.647086    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 03:13:59.647300 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:59.647133    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 03:13:59.647300 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:59.647270    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/220034f7-6232-4501-8842-f79fd919fc3c-kube-rbac-proxy-cm podName:220034f7-6232-4501-8842-f79fd919fc3c nodeName:}" failed. No retries permitted until 2026-08-04 03:14:03.647248901 +0000 UTC m=+274.327036001 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/220034f7-6232-4501-8842-f79fd919fc3c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rzf5k" (UID: "220034f7-6232-4501-8842-f79fd919fc3c") : configmap "kube-rbac-proxy" not found
Aug 04 03:13:59.647423 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:13:59.647312    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220034f7-6232-4501-8842-f79fd919fc3c-insights-runtime-extractor-tls podName:220034f7-6232-4501-8842-f79fd919fc3c nodeName:}" failed. No retries permitted until 2026-08-04 03:14:03.647293695 +0000 UTC m=+274.327080797 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/220034f7-6232-4501-8842-f79fd919fc3c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rzf5k" (UID: "220034f7-6232-4501-8842-f79fd919fc3c") : secret "insights-runtime-extractor-tls" not found
Aug 04 03:13:59.942373 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:13:59.942291    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ztnbv_fc5bd55b-006c-482c-b501-11c7e4a50629/serve-healthcheck-canary/0.log"
Aug 04 03:14:03.680994 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:03.680945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/220034f7-6232-4501-8842-f79fd919fc3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:14:03.681361 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:03.681020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/220034f7-6232-4501-8842-f79fd919fc3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:14:03.681361 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:14:03.681102    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 03:14:03.681361 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:14:03.681115    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 03:14:03.681361 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:14:03.681172    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220034f7-6232-4501-8842-f79fd919fc3c-insights-runtime-extractor-tls podName:220034f7-6232-4501-8842-f79fd919fc3c nodeName:}" failed. No retries permitted until 2026-08-04 03:14:11.681151764 +0000 UTC m=+282.360938861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/220034f7-6232-4501-8842-f79fd919fc3c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rzf5k" (UID: "220034f7-6232-4501-8842-f79fd919fc3c") : secret "insights-runtime-extractor-tls" not found
Aug 04 03:14:03.681361 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:14:03.681191    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/220034f7-6232-4501-8842-f79fd919fc3c-kube-rbac-proxy-cm podName:220034f7-6232-4501-8842-f79fd919fc3c nodeName:}" failed. No retries permitted until 2026-08-04 03:14:11.681182766 +0000 UTC m=+282.360969863 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/220034f7-6232-4501-8842-f79fd919fc3c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rzf5k" (UID: "220034f7-6232-4501-8842-f79fd919fc3c") : configmap "kube-rbac-proxy" not found
Aug 04 03:14:07.014679 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:07.014638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-nj2gk\" (UID: \"2b1f4754-8f2a-4da5-96db-db8a6b6b88fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk"
Aug 04 03:14:07.015085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:07.014718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-qgpdb\" (UID: \"f2d6c41f-c453-40d7-be08-ea72a2fcf37d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb"
Aug 04 03:14:07.017405 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:07.017377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2d6c41f-c453-40d7-be08-ea72a2fcf37d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-qgpdb\" (UID: \"f2d6c41f-c453-40d7-be08-ea72a2fcf37d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb"
Aug 04 03:14:07.017501 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:07.017437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b1f4754-8f2a-4da5-96db-db8a6b6b88fe-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-nj2gk\" (UID: \"2b1f4754-8f2a-4da5-96db-db8a6b6b88fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk"
Aug 04 03:14:07.081754 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:07.081715    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb"
Aug 04 03:14:07.092891 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:07.092768    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk"
Aug 04 03:14:07.225519 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:07.225460    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-gzkwr" podStartSLOduration=8.949327801 podStartE2EDuration="10.225439942s" totalImagesPullingTime="1.276112141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:13:59.595664294 +0000 UTC m=+270.275451414" watchObservedRunningTime="2026-08-04 03:14:07.225439942 +0000 UTC m=+277.905227066"
Aug 04 03:14:07.226249 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:07.226192    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb"]
Aug 04 03:14:07.228722 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:14:07.228693    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d6c41f_c453_40d7_be08_ea72a2fcf37d.slice/crio-c6f4ee51a7bb65dd252739a986b0cfea35eb69c85c3e11ce5e790461d4095a08 WatchSource:0}: Error finding container c6f4ee51a7bb65dd252739a986b0cfea35eb69c85c3e11ce5e790461d4095a08: Status 404 returned error can't find the container with id c6f4ee51a7bb65dd252739a986b0cfea35eb69c85c3e11ce5e790461d4095a08
Aug 04 03:14:07.242259 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:07.242236    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk"]
Aug 04 03:14:07.600101 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:07.600019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb" event={"ID":"f2d6c41f-c453-40d7-be08-ea72a2fcf37d","Type":"ContainerStarted","Data":"c6f4ee51a7bb65dd252739a986b0cfea35eb69c85c3e11ce5e790461d4095a08"}
Aug 04 03:14:07.600985 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:07.600953    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk" event={"ID":"2b1f4754-8f2a-4da5-96db-db8a6b6b88fe","Type":"ContainerStarted","Data":"de2d4557678b4ed2c019427c86612d704b69886aaf09c279c07db9bdd9f7f39e"}
Aug 04 03:14:09.610192 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:09.610021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk" event={"ID":"2b1f4754-8f2a-4da5-96db-db8a6b6b88fe","Type":"ContainerStarted","Data":"b5a2ad072f390a15b5074d4dc4e38e75045327111c22d333ad0adda4e03423cf"}
Aug 04 03:14:09.611673 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:09.611589    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb" event={"ID":"f2d6c41f-c453-40d7-be08-ea72a2fcf37d","Type":"ContainerStarted","Data":"338a8929b68eee6f03fa757c6b72c09099532522d94f53eb09f384bc1d62f668"}
Aug 04 03:14:09.631469 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:09.631396    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-qgpdb" podStartSLOduration=16.375336612 podStartE2EDuration="18.63137476s" totalImagesPullingTime="2.256038148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:09.630517772 +0000 UTC m=+280.310304941" watchObservedRunningTime="2026-08-04 03:14:09.63137476 +0000 UTC m=+280.311161879"
Aug 04 03:14:10.616135 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:10.616089    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk" event={"ID":"2b1f4754-8f2a-4da5-96db-db8a6b6b88fe","Type":"ContainerStarted","Data":"ef50dd05f558cbd94f4ceeee56b76f937a6ef23572d8797747b5acf81cbb7a5b"}
Aug 04 03:14:10.632259 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:10.632212    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nj2gk" podStartSLOduration=17.429188312 podStartE2EDuration="19.6321974s" totalImagesPullingTime="2.203009088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:10.630644506 +0000 UTC m=+281.310431625" watchObservedRunningTime="2026-08-04 03:14:10.6321974 +0000 UTC m=+281.311984518"
Aug 04 03:14:11.296274 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:14:11.296237    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv" podUID="9c78063e-e4f9-446b-90c5-89f0f2ddc001"
Aug 04 03:14:11.296274 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:14:11.296238    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-np56j" podUID="7b43cf1f-077d-485b-8a0d-fdec861aff43"
Aug 04 03:14:11.296274 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:14:11.296240    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz" podUID="5f607efb-3bb4-4755-a366-4d8b0c5aa431"
Aug 04 03:14:11.618900 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:11.618820    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"
Aug 04 03:14:11.618900 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:11.618841    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-np56j"
Aug 04 03:14:11.619398 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:11.618936    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:14:11.756915 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:11.756866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/220034f7-6232-4501-8842-f79fd919fc3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:14:11.757259 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:11.757229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/220034f7-6232-4501-8842-f79fd919fc3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:14:11.757805 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:11.757782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/220034f7-6232-4501-8842-f79fd919fc3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:14:11.759237 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:11.759220    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/220034f7-6232-4501-8842-f79fd919fc3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzf5k\" (UID: \"220034f7-6232-4501-8842-f79fd919fc3c\") " pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:14:11.874893 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:11.874802    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rzf5k"
Aug 04 03:14:11.998913 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:11.998886    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rzf5k"]
Aug 04 03:14:12.001352 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:14:12.001326    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220034f7_6232_4501_8842_f79fd919fc3c.slice/crio-06fda358f6bdf34fa1422707e49368e891f0ca90b8a36699d8ef4e70d05d4673 WatchSource:0}: Error finding container 06fda358f6bdf34fa1422707e49368e891f0ca90b8a36699d8ef4e70d05d4673: Status 404 returned error can't find the container with id 06fda358f6bdf34fa1422707e49368e891f0ca90b8a36699d8ef4e70d05d4673
Aug 04 03:14:12.622949 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:12.622919    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rzf5k" event={"ID":"220034f7-6232-4501-8842-f79fd919fc3c","Type":"ContainerStarted","Data":"0797f5491d172ba80044714427d44f37c0cb4a096a196f34055977b47df4bb0e"}
Aug 04 03:14:12.623327 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:12.622986    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rzf5k" event={"ID":"220034f7-6232-4501-8842-f79fd919fc3c","Type":"ContainerStarted","Data":"06fda358f6bdf34fa1422707e49368e891f0ca90b8a36699d8ef4e70d05d4673"}
Aug 04 03:14:13.628605 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:13.628566    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rzf5k" event={"ID":"220034f7-6232-4501-8842-f79fd919fc3c","Type":"ContainerStarted","Data":"cea6995802e88c39810334e20b055e498e479db7faa0a85ec02bb76d107ad309"}
Aug 04 03:14:14.633050 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:14.633017    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rzf5k" event={"ID":"220034f7-6232-4501-8842-f79fd919fc3c","Type":"ContainerStarted","Data":"17a07e8b4fad848d1fba32ae724c535a31fc66476d482db96d1c809e904974c1"}
Aug 04 03:14:14.650460 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:14.650411    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rzf5k" podStartSLOduration=17.786657056 podStartE2EDuration="19.650395182s" totalImagesPullingTime="1.863738126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:13:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:14.649059391 +0000 UTC m=+285.328846532" watchObservedRunningTime="2026-08-04 03:14:14.650395182 +0000 UTC m=+285.330182358"
Aug 04 03:14:15.181629 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.181577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:14:15.181629 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.181646    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-2t9qv\" (UID: \"9c78063e-e4f9-446b-90c5-89f0f2ddc001\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"
Aug 04 03:14:15.184158 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.184127    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls\") pod \"image-registry-958dbfdbb-wz2mz\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") " pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:14:15.184263 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.184164    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c78063e-e4f9-446b-90c5-89f0f2ddc001-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-2t9qv\" (UID: \"9c78063e-e4f9-446b-90c5-89f0f2ddc001\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"
Aug 04 03:14:15.222131 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.222102    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9lvw5\""
Aug 04 03:14:15.223110 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.223096    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-g7q88\""
Aug 04 03:14:15.230006 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.229990    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"
Aug 04 03:14:15.230058 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.230008    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:14:15.283522 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.283257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:14:15.286637 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.286609    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b43cf1f-077d-485b-8a0d-fdec861aff43-metrics-tls\") pod \"dns-default-np56j\" (UID: \"7b43cf1f-077d-485b-8a0d-fdec861aff43\") " pod="openshift-dns/dns-default-np56j"
Aug 04 03:14:15.382995 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.382941    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv"]
Aug 04 03:14:15.384699 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:14:15.384670    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c78063e_e4f9_446b_90c5_89f0f2ddc001.slice/crio-75e45a384073e92ec255b2cdca85f323cd3c0299facb5c3b1e6bb1405b895ee3 WatchSource:0}: Error finding container 75e45a384073e92ec255b2cdca85f323cd3c0299facb5c3b1e6bb1405b895ee3: Status 404 returned error can't find the container with id 75e45a384073e92ec255b2cdca85f323cd3c0299facb5c3b1e6bb1405b895ee3
Aug 04 03:14:15.401624 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.401600    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-958dbfdbb-wz2mz"]
Aug 04 03:14:15.405514 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:14:15.405488    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f607efb_3bb4_4755_a366_4d8b0c5aa431.slice/crio-607986b7ec77901dcf2b8eb59b54c6ac15f7c8343520a214e51bf431cf233d3b WatchSource:0}: Error finding container 607986b7ec77901dcf2b8eb59b54c6ac15f7c8343520a214e51bf431cf233d3b: Status 404 returned error can't find the container with id 607986b7ec77901dcf2b8eb59b54c6ac15f7c8343520a214e51bf431cf233d3b
Aug 04 03:14:15.522263 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.522233    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4d7vw\""
Aug 04 03:14:15.530115 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.530097    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-np56j"
Aug 04 03:14:15.639462 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.639425    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz" event={"ID":"5f607efb-3bb4-4755-a366-4d8b0c5aa431","Type":"ContainerStarted","Data":"604abb67b072cde6693e803e1203aab19013f4be7a2529c8fcd99be0a5f90ab9"}
Aug 04 03:14:15.639879 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.639485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz" event={"ID":"5f607efb-3bb4-4755-a366-4d8b0c5aa431","Type":"ContainerStarted","Data":"607986b7ec77901dcf2b8eb59b54c6ac15f7c8343520a214e51bf431cf233d3b"}
Aug 04 03:14:15.639879 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.639515    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:14:15.640659 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.640630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv" event={"ID":"9c78063e-e4f9-446b-90c5-89f0f2ddc001","Type":"ContainerStarted","Data":"75e45a384073e92ec255b2cdca85f323cd3c0299facb5c3b1e6bb1405b895ee3"}
Aug 04 03:14:15.658693 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.658646    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz" podStartSLOduration=263.65863079 podStartE2EDuration="4m23.65863079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:15.657416482 +0000 UTC m=+286.337203624" watchObservedRunningTime="2026-08-04 03:14:15.65863079 +0000 UTC m=+286.338417909"
Aug 04 03:14:15.662528 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:15.662507    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-np56j"]
Aug 04 03:14:15.665017 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:14:15.664991    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b43cf1f_077d_485b_8a0d_fdec861aff43.slice/crio-1ba13b74424a474ecc1ac968db08f4735d325ca727e30b6499f2f4cab22d7cdb WatchSource:0}: Error finding container 1ba13b74424a474ecc1ac968db08f4735d325ca727e30b6499f2f4cab22d7cdb: Status 404 returned error can't find the container with id 1ba13b74424a474ecc1ac968db08f4735d325ca727e30b6499f2f4cab22d7cdb
Aug 04 03:14:16.646610 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:16.646260    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-np56j" event={"ID":"7b43cf1f-077d-485b-8a0d-fdec861aff43","Type":"ContainerStarted","Data":"1ba13b74424a474ecc1ac968db08f4735d325ca727e30b6499f2f4cab22d7cdb"}
Aug 04 03:14:17.651092 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:17.651052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv" event={"ID":"9c78063e-e4f9-446b-90c5-89f0f2ddc001","Type":"ContainerStarted","Data":"e133c62dae27328947315d933abcfcbf1e08a9d5cede8e8df07848b3f01e8dec"}
Aug 04 03:14:17.652893 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:17.652872    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-np56j" event={"ID":"7b43cf1f-077d-485b-8a0d-fdec861aff43","Type":"ContainerStarted","Data":"c9d3db07b2fe3bf5312ae0e35f9d35f68e7b8944406a278b85ce805837cc0106"}
Aug 04 03:14:17.653014 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:17.652911    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-np56j" event={"ID":"7b43cf1f-077d-485b-8a0d-fdec861aff43","Type":"ContainerStarted","Data":"bfb3db3a8bc916c9d951d3820787a4481045b2b44b6d897d8f8f2ba968c7e88a"}
Aug 04 03:14:17.653014 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:17.652937    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-np56j"
Aug 04 03:14:17.666451 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:17.666378    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-2t9qv" podStartSLOduration=266.219379309 podStartE2EDuration="4m27.666366026s" totalImagesPullingTime="1.446986717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:17.665454268 +0000 UTC m=+288.345241413" watchObservedRunningTime="2026-08-04 03:14:17.666366026 +0000 UTC m=+288.346153146"
Aug 04 03:14:17.683485 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:17.683448    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-np56j" podStartSLOduration=250.932634594 podStartE2EDuration="4m12.683439049s" totalImagesPullingTime="1.750804455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:17.683154738 +0000 UTC m=+288.362941858" watchObservedRunningTime="2026-08-04 03:14:17.683439049 +0000 UTC m=+288.363226168"
Aug 04 03:14:19.748400 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:19.748364    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9x8zj"]
Aug 04 03:14:19.751614 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:19.751587    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9x8zj"
Aug 04 03:14:19.758919 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:19.758900    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mnnps\""
Aug 04 03:14:19.759799 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:19.759784    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Aug 04 03:14:19.772809 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:19.772780    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9x8zj"]
Aug 04 03:14:19.921093 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:19.921060    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/57105572-98a4-43d9-98c7-ba99c117ad2e-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-9x8zj\" (UID: \"57105572-98a4-43d9-98c7-ba99c117ad2e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9x8zj"
Aug 04 03:14:20.021694 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:20.021614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/57105572-98a4-43d9-98c7-ba99c117ad2e-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-9x8zj\" (UID: \"57105572-98a4-43d9-98c7-ba99c117ad2e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9x8zj"
Aug 04 03:14:20.023933 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:20.023910    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/57105572-98a4-43d9-98c7-ba99c117ad2e-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-9x8zj\" (UID: \"57105572-98a4-43d9-98c7-ba99c117ad2e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9x8zj"
Aug 04 03:14:20.059954 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:20.059926    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9x8zj"
Aug 04 03:14:20.184386 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:20.184355    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9x8zj"]
Aug 04 03:14:20.187136 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:14:20.187111    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57105572_98a4_43d9_98c7_ba99c117ad2e.slice/crio-c6da470213b249162dc29d89fdee5c76feedc46077696b4864893e88483879dd WatchSource:0}: Error finding container c6da470213b249162dc29d89fdee5c76feedc46077696b4864893e88483879dd: Status 404 returned error can't find the container with id c6da470213b249162dc29d89fdee5c76feedc46077696b4864893e88483879dd
Aug 04 03:14:20.666468 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:20.666436    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9x8zj" event={"ID":"57105572-98a4-43d9-98c7-ba99c117ad2e","Type":"ContainerStarted","Data":"c6da470213b249162dc29d89fdee5c76feedc46077696b4864893e88483879dd"}
Aug 04 03:14:21.076828 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.076793    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fb5db745b-p9pfz"]
Aug 04 03:14:21.080517 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.080494    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.083225 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.083185    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Aug 04 03:14:21.084222 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.084199    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Aug 04 03:14:21.084346 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.084240    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Aug 04 03:14:21.084507 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.084490    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Aug 04 03:14:21.084573 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.084564    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Aug 04 03:14:21.084629 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.084584    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Aug 04 03:14:21.084687 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.084624    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wwf2m\""
Aug 04 03:14:21.084687 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.084490    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Aug 04 03:14:21.088800 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.088745    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fb5db745b-p9pfz"]
Aug 04 03:14:21.233912 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.233875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3782a733-7495-46a9-bb37-acf3df0fb88f-oauth-serving-cert\") pod \"console-5fb5db745b-p9pfz\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") " pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.234124 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.233946    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3782a733-7495-46a9-bb37-acf3df0fb88f-console-oauth-config\") pod \"console-5fb5db745b-p9pfz\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") " pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.234124 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.234063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3782a733-7495-46a9-bb37-acf3df0fb88f-service-ca\") pod \"console-5fb5db745b-p9pfz\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") " pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.234124 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.234098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfjd5\" (UniqueName: \"kubernetes.io/projected/3782a733-7495-46a9-bb37-acf3df0fb88f-kube-api-access-cfjd5\") pod \"console-5fb5db745b-p9pfz\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") " pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.234264 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.234146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3782a733-7495-46a9-bb37-acf3df0fb88f-console-serving-cert\") pod \"console-5fb5db745b-p9pfz\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") " pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.234264 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.234178    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3782a733-7495-46a9-bb37-acf3df0fb88f-console-config\") pod \"console-5fb5db745b-p9pfz\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") " pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.334859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.334836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3782a733-7495-46a9-bb37-acf3df0fb88f-service-ca\") pod \"console-5fb5db745b-p9pfz\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") " pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.334993 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.334869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfjd5\" (UniqueName: \"kubernetes.io/projected/3782a733-7495-46a9-bb37-acf3df0fb88f-kube-api-access-cfjd5\") pod \"console-5fb5db745b-p9pfz\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") " pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.334993 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.334912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3782a733-7495-46a9-bb37-acf3df0fb88f-console-serving-cert\") pod \"console-5fb5db745b-p9pfz\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") " pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.334993 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.334943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3782a733-7495-46a9-bb37-acf3df0fb88f-console-config\") pod \"console-5fb5db745b-p9pfz\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") " pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.335162 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.334999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3782a733-7495-46a9-bb37-acf3df0fb88f-oauth-serving-cert\") pod \"console-5fb5db745b-p9pfz\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") " pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.335162 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.335057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3782a733-7495-46a9-bb37-acf3df0fb88f-console-oauth-config\") pod \"console-5fb5db745b-p9pfz\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") " pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.335622 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.335594    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3782a733-7495-46a9-bb37-acf3df0fb88f-console-config\") pod \"console-5fb5db745b-p9pfz\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") " pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.335733 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.335594    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3782a733-7495-46a9-bb37-acf3df0fb88f-service-ca\") pod \"console-5fb5db745b-p9pfz\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") " pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.335733 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.335686    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3782a733-7495-46a9-bb37-acf3df0fb88f-oauth-serving-cert\") pod \"console-5fb5db745b-p9pfz\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") " pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.337352 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.337333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3782a733-7495-46a9-bb37-acf3df0fb88f-console-serving-cert\") pod \"console-5fb5db745b-p9pfz\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") " pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.337536 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.337516    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3782a733-7495-46a9-bb37-acf3df0fb88f-console-oauth-config\") pod \"console-5fb5db745b-p9pfz\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") " pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.342420 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.342397    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfjd5\" (UniqueName: \"kubernetes.io/projected/3782a733-7495-46a9-bb37-acf3df0fb88f-kube-api-access-cfjd5\") pod \"console-5fb5db745b-p9pfz\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") " pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.392546 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.392520    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:21.516784 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.516750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fb5db745b-p9pfz"]
Aug 04 03:14:21.520558 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:14:21.520527    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3782a733_7495_46a9_bb37_acf3df0fb88f.slice/crio-fd6a9af02fd04b7b18e0abb83f76141b336aa5e616c4b131d8c5225e33e8aa0a WatchSource:0}: Error finding container fd6a9af02fd04b7b18e0abb83f76141b336aa5e616c4b131d8c5225e33e8aa0a: Status 404 returned error can't find the container with id fd6a9af02fd04b7b18e0abb83f76141b336aa5e616c4b131d8c5225e33e8aa0a
Aug 04 03:14:21.670699 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.670606    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9x8zj" event={"ID":"57105572-98a4-43d9-98c7-ba99c117ad2e","Type":"ContainerStarted","Data":"e41ab92ed38e8c7c14782c81e8bb5302fa64778011d9ce2452648e43e2b635a4"}
Aug 04 03:14:21.670699 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.670671    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9x8zj"
Aug 04 03:14:21.671769 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.671744    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fb5db745b-p9pfz" event={"ID":"3782a733-7495-46a9-bb37-acf3df0fb88f","Type":"ContainerStarted","Data":"fd6a9af02fd04b7b18e0abb83f76141b336aa5e616c4b131d8c5225e33e8aa0a"}
Aug 04 03:14:21.675291 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.675273    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9x8zj"
Aug 04 03:14:21.685405 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:21.685370    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9x8zj" podStartSLOduration=1.571456874 podStartE2EDuration="2.685358461s" totalImagesPullingTime="1.113901587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:21.684063107 +0000 UTC m=+292.363850256" watchObservedRunningTime="2026-08-04 03:14:21.685358461 +0000 UTC m=+292.365145579"
Aug 04 03:14:23.840255 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:14:23.840206    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-c5d74" podUID="b39560be-b9af-4ff7-8a5b-7882c66a03d8"
Aug 04 03:14:24.685178 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:24.685097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fb5db745b-p9pfz" event={"ID":"3782a733-7495-46a9-bb37-acf3df0fb88f","Type":"ContainerStarted","Data":"3b51df457481f04585e8be97e2d20e1d35a968f48393b6a4a43a02cfd15b71ca"}
Aug 04 03:14:24.703301 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:24.703259    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fb5db745b-p9pfz" podStartSLOduration=1.004739621 podStartE2EDuration="3.703244222s" totalImagesPullingTime="2.698504601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:24.701450461 +0000 UTC m=+295.381237598" watchObservedRunningTime="2026-08-04 03:14:24.703244222 +0000 UTC m=+295.383031415"
Aug 04 03:14:27.382481 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.382450    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"]
Aug 04 03:14:27.388036 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.388014    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:27.390533 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.390506    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Aug 04 03:14:27.391380 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.391359    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Aug 04 03:14:27.391496 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.391402    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2wqcz\""
Aug 04 03:14:27.391555 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.391504    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Aug 04 03:14:27.391617 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.391603    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Aug 04 03:14:27.396694 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.396678    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xmjtq"]
Aug 04 03:14:27.400683 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.400663    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"]
Aug 04 03:14:27.400813 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.400796    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.403088 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.403070    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Aug 04 03:14:27.403227 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.403209    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h8mnb\""
Aug 04 03:14:27.403689 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.403531    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Aug 04 03:14:27.404057 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.403835    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Aug 04 03:14:27.485327 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.485286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/437947e9-5b4e-4466-ba95-79d38416d1fd-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-frz48\" (UID: \"437947e9-5b4e-4466-ba95-79d38416d1fd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:27.485542 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.485340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtx84\" (UniqueName: \"kubernetes.io/projected/437947e9-5b4e-4466-ba95-79d38416d1fd-kube-api-access-gtx84\") pod \"kube-state-metrics-6bd9664c4-frz48\" (UID: \"437947e9-5b4e-4466-ba95-79d38416d1fd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:27.485542 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.485421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/437947e9-5b4e-4466-ba95-79d38416d1fd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-frz48\" (UID: \"437947e9-5b4e-4466-ba95-79d38416d1fd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:27.485542 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.485451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/437947e9-5b4e-4466-ba95-79d38416d1fd-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-frz48\" (UID: \"437947e9-5b4e-4466-ba95-79d38416d1fd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:27.485542 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.485489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/437947e9-5b4e-4466-ba95-79d38416d1fd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-frz48\" (UID: \"437947e9-5b4e-4466-ba95-79d38416d1fd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:27.485756 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.485570    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/437947e9-5b4e-4466-ba95-79d38416d1fd-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-frz48\" (UID: \"437947e9-5b4e-4466-ba95-79d38416d1fd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:27.586679 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.586646    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtx84\" (UniqueName: \"kubernetes.io/projected/437947e9-5b4e-4466-ba95-79d38416d1fd-kube-api-access-gtx84\") pod \"kube-state-metrics-6bd9664c4-frz48\" (UID: \"437947e9-5b4e-4466-ba95-79d38416d1fd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:27.587058 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.587033    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8f46088b-d589-4008-ad8c-56c451ebb5b4-node-exporter-textfile\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.587249 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.587233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8f46088b-d589-4008-ad8c-56c451ebb5b4-node-exporter-wtmp\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.587401 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.587353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/437947e9-5b4e-4466-ba95-79d38416d1fd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-frz48\" (UID: \"437947e9-5b4e-4466-ba95-79d38416d1fd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:27.587530 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.587406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8f46088b-d589-4008-ad8c-56c451ebb5b4-node-exporter-accelerators-collector-config\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.587530 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.587436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/437947e9-5b4e-4466-ba95-79d38416d1fd-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-frz48\" (UID: \"437947e9-5b4e-4466-ba95-79d38416d1fd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:27.587530 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.587505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8f46088b-d589-4008-ad8c-56c451ebb5b4-root\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.587685 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.587540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/437947e9-5b4e-4466-ba95-79d38416d1fd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-frz48\" (UID: \"437947e9-5b4e-4466-ba95-79d38416d1fd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:27.587685 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.587571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f46088b-d589-4008-ad8c-56c451ebb5b4-metrics-client-ca\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.587685 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.587613    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f46088b-d589-4008-ad8c-56c451ebb5b4-sys\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.587685 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.587639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfn55\" (UniqueName: \"kubernetes.io/projected/8f46088b-d589-4008-ad8c-56c451ebb5b4-kube-api-access-gfn55\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.587685 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.587673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/437947e9-5b4e-4466-ba95-79d38416d1fd-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-frz48\" (UID: \"437947e9-5b4e-4466-ba95-79d38416d1fd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:27.587918 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.587704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/437947e9-5b4e-4466-ba95-79d38416d1fd-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-frz48\" (UID: \"437947e9-5b4e-4466-ba95-79d38416d1fd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:27.587918 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.587746    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f46088b-d589-4008-ad8c-56c451ebb5b4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.587918 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.587774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8f46088b-d589-4008-ad8c-56c451ebb5b4-node-exporter-tls\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.588257 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:14:27.588237    2579 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Aug 04 03:14:27.588404 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:14:27.588390    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/437947e9-5b4e-4466-ba95-79d38416d1fd-kube-state-metrics-tls podName:437947e9-5b4e-4466-ba95-79d38416d1fd nodeName:}" failed. No retries permitted until 2026-08-04 03:14:28.088368362 +0000 UTC m=+298.768155470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/437947e9-5b4e-4466-ba95-79d38416d1fd-kube-state-metrics-tls") pod "kube-state-metrics-6bd9664c4-frz48" (UID: "437947e9-5b4e-4466-ba95-79d38416d1fd") : secret "kube-state-metrics-tls" not found
Aug 04 03:14:27.588507 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.588485    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/437947e9-5b4e-4466-ba95-79d38416d1fd-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-frz48\" (UID: \"437947e9-5b4e-4466-ba95-79d38416d1fd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:27.589297 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.588877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/437947e9-5b4e-4466-ba95-79d38416d1fd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-frz48\" (UID: \"437947e9-5b4e-4466-ba95-79d38416d1fd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:27.589422 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.589237    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/437947e9-5b4e-4466-ba95-79d38416d1fd-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-frz48\" (UID: \"437947e9-5b4e-4466-ba95-79d38416d1fd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:27.591304 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.591280    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/437947e9-5b4e-4466-ba95-79d38416d1fd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-frz48\" (UID: \"437947e9-5b4e-4466-ba95-79d38416d1fd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:27.599684 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.599653    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtx84\" (UniqueName: \"kubernetes.io/projected/437947e9-5b4e-4466-ba95-79d38416d1fd-kube-api-access-gtx84\") pod \"kube-state-metrics-6bd9664c4-frz48\" (UID: \"437947e9-5b4e-4466-ba95-79d38416d1fd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:27.659015 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.658919    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-np56j"
Aug 04 03:14:27.688753 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.688717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8f46088b-d589-4008-ad8c-56c451ebb5b4-root\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.688932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.688769    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f46088b-d589-4008-ad8c-56c451ebb5b4-metrics-client-ca\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.688932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.688805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f46088b-d589-4008-ad8c-56c451ebb5b4-sys\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.688932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.688830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfn55\" (UniqueName: \"kubernetes.io/projected/8f46088b-d589-4008-ad8c-56c451ebb5b4-kube-api-access-gfn55\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.688932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.688849    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8f46088b-d589-4008-ad8c-56c451ebb5b4-root\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.688932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.688875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f46088b-d589-4008-ad8c-56c451ebb5b4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.688932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.688901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8f46088b-d589-4008-ad8c-56c451ebb5b4-node-exporter-tls\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.688932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.688922    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f46088b-d589-4008-ad8c-56c451ebb5b4-sys\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.689308 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.688941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8f46088b-d589-4008-ad8c-56c451ebb5b4-node-exporter-textfile\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.689308 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.688995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8f46088b-d589-4008-ad8c-56c451ebb5b4-node-exporter-wtmp\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.689308 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.689027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8f46088b-d589-4008-ad8c-56c451ebb5b4-node-exporter-accelerators-collector-config\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.689618 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.689598    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f46088b-d589-4008-ad8c-56c451ebb5b4-metrics-client-ca\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.689782 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.689623    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8f46088b-d589-4008-ad8c-56c451ebb5b4-node-exporter-accelerators-collector-config\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.689980 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:14:27.689950    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Aug 04 03:14:27.690140 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:14:27.690117    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f46088b-d589-4008-ad8c-56c451ebb5b4-node-exporter-tls podName:8f46088b-d589-4008-ad8c-56c451ebb5b4 nodeName:}" failed. No retries permitted until 2026-08-04 03:14:28.190099886 +0000 UTC m=+298.869886988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8f46088b-d589-4008-ad8c-56c451ebb5b4-node-exporter-tls") pod "node-exporter-xmjtq" (UID: "8f46088b-d589-4008-ad8c-56c451ebb5b4") : secret "node-exporter-tls" not found
Aug 04 03:14:27.690233 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.690161    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8f46088b-d589-4008-ad8c-56c451ebb5b4-node-exporter-textfile\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.690315 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.690029    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8f46088b-d589-4008-ad8c-56c451ebb5b4-node-exporter-wtmp\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.692890 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.692836    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f46088b-d589-4008-ad8c-56c451ebb5b4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:27.697608 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:27.697587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfn55\" (UniqueName: \"kubernetes.io/projected/8f46088b-d589-4008-ad8c-56c451ebb5b4-kube-api-access-gfn55\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:28.093160 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.093118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/437947e9-5b4e-4466-ba95-79d38416d1fd-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-frz48\" (UID: \"437947e9-5b4e-4466-ba95-79d38416d1fd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:28.095704 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.095679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/437947e9-5b4e-4466-ba95-79d38416d1fd-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-frz48\" (UID: \"437947e9-5b4e-4466-ba95-79d38416d1fd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:28.194038 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.194003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8f46088b-d589-4008-ad8c-56c451ebb5b4-node-exporter-tls\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:28.196286 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.196259    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8f46088b-d589-4008-ad8c-56c451ebb5b4-node-exporter-tls\") pod \"node-exporter-xmjtq\" (UID: \"8f46088b-d589-4008-ad8c-56c451ebb5b4\") " pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:28.298934 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.298903    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"
Aug 04 03:14:28.312384 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.312353    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xmjtq"
Aug 04 03:14:28.327075 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:14:28.327029    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f46088b_d589_4008_ad8c_56c451ebb5b4.slice/crio-cb9140c07b9b6effdf5864e05f733560e846ea96a425f9c1fc96c354afcf23fc WatchSource:0}: Error finding container cb9140c07b9b6effdf5864e05f733560e846ea96a425f9c1fc96c354afcf23fc: Status 404 returned error can't find the container with id cb9140c07b9b6effdf5864e05f733560e846ea96a425f9c1fc96c354afcf23fc
Aug 04 03:14:28.427304 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.427218    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 03:14:28.433121 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.433094    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.436791 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.435767    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Aug 04 03:14:28.436791 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.435887    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Aug 04 03:14:28.436791 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.436131    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Aug 04 03:14:28.436791 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.436142    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Aug 04 03:14:28.436791 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.436336    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Aug 04 03:14:28.436791 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.436349    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Aug 04 03:14:28.436791 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.436503    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pmm9s\""
Aug 04 03:14:28.436791 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.436596    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Aug 04 03:14:28.436791 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.436705    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Aug 04 03:14:28.437825 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.437407    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Aug 04 03:14:28.446492 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.446457    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 03:14:28.461811 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.461785    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-frz48"]
Aug 04 03:14:28.466687 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:14:28.466655    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod437947e9_5b4e_4466_ba95_79d38416d1fd.slice/crio-3f4590cf99c0f17f80ab0e0fe2cbf75864ce7dc7696c67251e4b61dd9ffbafde WatchSource:0}: Error finding container 3f4590cf99c0f17f80ab0e0fe2cbf75864ce7dc7696c67251e4b61dd9ffbafde: Status 404 returned error can't find the container with id 3f4590cf99c0f17f80ab0e0fe2cbf75864ce7dc7696c67251e4b61dd9ffbafde
Aug 04 03:14:28.598613 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.598575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-web-config\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.598768 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.598624    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9353a0a-4f31-42c2-bbbf-b623049e89c3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.598768 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.598728    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9353a0a-4f31-42c2-bbbf-b623049e89c3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.598768 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.598761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccnnc\" (UniqueName: \"kubernetes.io/projected/f9353a0a-4f31-42c2-bbbf-b623049e89c3-kube-api-access-ccnnc\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.598949 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.598788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-config-volume\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.598949 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.598819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.598949 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.598837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9353a0a-4f31-42c2-bbbf-b623049e89c3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.598949 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.598935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.599157 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.598989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9353a0a-4f31-42c2-bbbf-b623049e89c3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.599157 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.599011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.599157 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.599030    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9353a0a-4f31-42c2-bbbf-b623049e89c3-config-out\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.599157 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.599052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.599157 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.599082    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.700129 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.700049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9353a0a-4f31-42c2-bbbf-b623049e89c3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.700129 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.700093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccnnc\" (UniqueName: \"kubernetes.io/projected/f9353a0a-4f31-42c2-bbbf-b623049e89c3-kube-api-access-ccnnc\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.700336 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.700217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-config-volume\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.700336 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.700247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.700336 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.700270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9353a0a-4f31-42c2-bbbf-b623049e89c3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.700336 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.700303    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.700538 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.700338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9353a0a-4f31-42c2-bbbf-b623049e89c3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.700538 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.700365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.700538 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.700392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9353a0a-4f31-42c2-bbbf-b623049e89c3-config-out\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.700538 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.700416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.700538 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.700440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.700775 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.700671    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9353a0a-4f31-42c2-bbbf-b623049e89c3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.701413 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.700891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-web-config\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.701413 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.700983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9353a0a-4f31-42c2-bbbf-b623049e89c3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.701413 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.700992    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9353a0a-4f31-42c2-bbbf-b623049e89c3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.702935 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.702880    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9353a0a-4f31-42c2-bbbf-b623049e89c3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.704068 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.703985    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xmjtq" event={"ID":"8f46088b-d589-4008-ad8c-56c451ebb5b4","Type":"ContainerStarted","Data":"cb9140c07b9b6effdf5864e05f733560e846ea96a425f9c1fc96c354afcf23fc"}
Aug 04 03:14:28.704068 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.704023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.704241 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.704116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.704241 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.704173    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.704496 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.704440    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.704947 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.704901    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9353a0a-4f31-42c2-bbbf-b623049e89c3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.705067 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.705031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-web-config\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.705728 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.705662    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48" event={"ID":"437947e9-5b4e-4466-ba95-79d38416d1fd","Type":"ContainerStarted","Data":"3f4590cf99c0f17f80ab0e0fe2cbf75864ce7dc7696c67251e4b61dd9ffbafde"}
Aug 04 03:14:28.705840 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.705818    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9353a0a-4f31-42c2-bbbf-b623049e89c3-config-out\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.706016 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.705994    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-config-volume\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.706089 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.706034    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.707661 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.707640    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccnnc\" (UniqueName: \"kubernetes.io/projected/f9353a0a-4f31-42c2-bbbf-b623049e89c3-kube-api-access-ccnnc\") pod \"alertmanager-main-0\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.746423 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.746391    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:14:28.911137 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:28.911040    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 03:14:28.913909 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:14:28.913878    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9353a0a_4f31_42c2_bbbf_b623049e89c3.slice/crio-7fabbba4c3e885e9c402a6c98707886826859f3ed8af1ae7c37a3477c472df30 WatchSource:0}: Error finding container 7fabbba4c3e885e9c402a6c98707886826859f3ed8af1ae7c37a3477c472df30: Status 404 returned error can't find the container with id 7fabbba4c3e885e9c402a6c98707886826859f3ed8af1ae7c37a3477c472df30
Aug 04 03:14:29.712604 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:29.712573    2579 generic.go:356] "Generic (PLEG): container finished" podID="8f46088b-d589-4008-ad8c-56c451ebb5b4" containerID="a25f9122373d76d094cb0d9028a8819a9ffff0a69fc7d0f29867bc937d638fee" exitCode=0
Aug 04 03:14:29.713472 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:29.712891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xmjtq" event={"ID":"8f46088b-d589-4008-ad8c-56c451ebb5b4","Type":"ContainerDied","Data":"a25f9122373d76d094cb0d9028a8819a9ffff0a69fc7d0f29867bc937d638fee"}
Aug 04 03:14:29.718445 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:29.717583    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9353a0a-4f31-42c2-bbbf-b623049e89c3","Type":"ContainerStarted","Data":"7fabbba4c3e885e9c402a6c98707886826859f3ed8af1ae7c37a3477c472df30"}
Aug 04 03:14:29.744244 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:29.744227    2579 kubelet.go:1648] "Image garbage collection succeeded"
Aug 04 03:14:30.723716 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:30.723678    2579 generic.go:356] "Generic (PLEG): container finished" podID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerID="15e6294804cbc205fdac5a04c756c971218d4a76b18293f295f3b108bfd5582f" exitCode=0
Aug 04 03:14:30.724060 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:30.723766    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9353a0a-4f31-42c2-bbbf-b623049e89c3","Type":"ContainerDied","Data":"15e6294804cbc205fdac5a04c756c971218d4a76b18293f295f3b108bfd5582f"}
Aug 04 03:14:30.728353 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:30.727767    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xmjtq" event={"ID":"8f46088b-d589-4008-ad8c-56c451ebb5b4","Type":"ContainerStarted","Data":"b91506a0580421df7a3f4ad4ed6ceecb87ec9ea225bfa31a820fd8b3051bbbb1"}
Aug 04 03:14:30.728353 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:30.727828    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xmjtq" event={"ID":"8f46088b-d589-4008-ad8c-56c451ebb5b4","Type":"ContainerStarted","Data":"bec249b362b518df7f0999e7a641173e6ec3c7a094b5054bd2ebe2686861b56f"}
Aug 04 03:14:30.730876 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:30.730755    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48" event={"ID":"437947e9-5b4e-4466-ba95-79d38416d1fd","Type":"ContainerStarted","Data":"951b96ecc3dd7447f9f2de0eca31f7f44f0e6f890ff45b95502950851ba8e77d"}
Aug 04 03:14:30.772904 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:30.772817    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xmjtq" podStartSLOduration=2.765722165 podStartE2EDuration="3.772800893s" totalImagesPullingTime="943.960331ms" totalInitContainerRuntime="63.118397ms" isStatefulPod=true podCreationTimestamp="2026-08-04 03:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:30.772116215 +0000 UTC m=+301.451903335" watchObservedRunningTime="2026-08-04 03:14:30.772800893 +0000 UTC m=+301.452588011"
Aug 04 03:14:31.393437 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.393396    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:31.393618 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.393462    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:31.398553 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.398526    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:31.462081 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.462050    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7966db58bc-xzdpq"]
Aug 04 03:14:31.465664 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.465635    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.476207 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.475615    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Aug 04 03:14:31.482072 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.482044    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7966db58bc-xzdpq"]
Aug 04 03:14:31.524604 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.524566    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-console-config\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.524793 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.524612    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-trusted-ca-bundle\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.524793 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.524733    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbhcs\" (UniqueName: \"kubernetes.io/projected/ead8ec72-b5e9-4496-8e09-f32638fdadfd-kube-api-access-cbhcs\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.524793 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.524783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ead8ec72-b5e9-4496-8e09-f32638fdadfd-console-serving-cert\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.524921 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.524845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-service-ca\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.524921 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.524893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-oauth-serving-cert\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.524921 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.524917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ead8ec72-b5e9-4496-8e09-f32638fdadfd-console-oauth-config\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.625946 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.625899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbhcs\" (UniqueName: \"kubernetes.io/projected/ead8ec72-b5e9-4496-8e09-f32638fdadfd-kube-api-access-cbhcs\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.626128 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.625978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ead8ec72-b5e9-4496-8e09-f32638fdadfd-console-serving-cert\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.626128 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.626020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-service-ca\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.626128 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.626060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-oauth-serving-cert\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.626128 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.626090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ead8ec72-b5e9-4496-8e09-f32638fdadfd-console-oauth-config\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.626360 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.626130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-console-config\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.626360 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.626162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-trusted-ca-bundle\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.627240 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.627187    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-trusted-ca-bundle\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.627240 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.627205    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-oauth-serving-cert\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.627240 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.627212    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-console-config\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.627240 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.627210    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-service-ca\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.629399 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.629377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ead8ec72-b5e9-4496-8e09-f32638fdadfd-console-oauth-config\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.629574 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.629550    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ead8ec72-b5e9-4496-8e09-f32638fdadfd-console-serving-cert\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.634033 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.634006    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbhcs\" (UniqueName: \"kubernetes.io/projected/ead8ec72-b5e9-4496-8e09-f32638fdadfd-kube-api-access-cbhcs\") pod \"console-7966db58bc-xzdpq\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") " pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.736197 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.736134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48" event={"ID":"437947e9-5b4e-4466-ba95-79d38416d1fd","Type":"ContainerStarted","Data":"e36b70564c63c5b496fee0e7fa8e2c13117420f1324b3d4057119b4c7df2f668"}
Aug 04 03:14:31.736683 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.736223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48" event={"ID":"437947e9-5b4e-4466-ba95-79d38416d1fd","Type":"ContainerStarted","Data":"47fd2d1aa46f160dae6ccde41d406d1ab625e0748fd7e280a081116df4d47b1d"}
Aug 04 03:14:31.740033 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.740009    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:14:31.755201 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.755147    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-frz48" podStartSLOduration=2.697169687 podStartE2EDuration="4.755130168s" totalImagesPullingTime="2.057960481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:31.753820461 +0000 UTC m=+302.433607580" watchObservedRunningTime="2026-08-04 03:14:31.755130168 +0000 UTC m=+302.434917352"
Aug 04 03:14:31.781190 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.781101    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:31.926683 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:31.926610    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7966db58bc-xzdpq"]
Aug 04 03:14:31.928924 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:14:31.928895    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead8ec72_b5e9_4496_8e09_f32638fdadfd.slice/crio-00742dfa79b09a690d4bb63fc417ed5cf10a998a927bfcb1a4f0c34288589663 WatchSource:0}: Error finding container 00742dfa79b09a690d4bb63fc417ed5cf10a998a927bfcb1a4f0c34288589663: Status 404 returned error can't find the container with id 00742dfa79b09a690d4bb63fc417ed5cf10a998a927bfcb1a4f0c34288589663
Aug 04 03:14:32.742705 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:32.742675    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9353a0a-4f31-42c2-bbbf-b623049e89c3","Type":"ContainerStarted","Data":"646d37c280d88d1c46b031ca16167c7a70671e0aff3472eb2ef046c53fb542f9"}
Aug 04 03:14:32.743136 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:32.742754    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9353a0a-4f31-42c2-bbbf-b623049e89c3","Type":"ContainerStarted","Data":"93dc098471b03a648ae9883e70586d0c801343c3e2a0ed5950048057f5e9b095"}
Aug 04 03:14:32.743136 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:32.742802    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9353a0a-4f31-42c2-bbbf-b623049e89c3","Type":"ContainerStarted","Data":"afea8032f4219653c0899bc3305723935ca338ea8d2d6a2ecf8abe996831c800"}
Aug 04 03:14:32.744143 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:32.744077    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7966db58bc-xzdpq" event={"ID":"ead8ec72-b5e9-4496-8e09-f32638fdadfd","Type":"ContainerStarted","Data":"e8c1e65638f4263494c60ec5d3cfe624d087e339d4661a4c51f739fe4ebba3c7"}
Aug 04 03:14:32.744143 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:32.744126    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7966db58bc-xzdpq" event={"ID":"ead8ec72-b5e9-4496-8e09-f32638fdadfd","Type":"ContainerStarted","Data":"00742dfa79b09a690d4bb63fc417ed5cf10a998a927bfcb1a4f0c34288589663"}
Aug 04 03:14:32.760296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:32.760258    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7966db58bc-xzdpq" podStartSLOduration=1.760244036 podStartE2EDuration="1.760244036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:32.75956972 +0000 UTC m=+303.439356878" watchObservedRunningTime="2026-08-04 03:14:32.760244036 +0000 UTC m=+303.440031154"
Aug 04 03:14:32.835245 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:32.835228    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 03:14:33.550284 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.550254    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 03:14:33.555044 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.554976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.557693 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.557522    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Aug 04 03:14:33.558354 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.557844    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Aug 04 03:14:33.558354 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.557881    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Aug 04 03:14:33.558354 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.558153    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Aug 04 03:14:33.558354 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.558164    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Aug 04 03:14:33.560164 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.558704    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Aug 04 03:14:33.560164 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.558932    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Aug 04 03:14:33.560164 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.558941    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6jj2s\""
Aug 04 03:14:33.560164 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.559190    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Aug 04 03:14:33.560164 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.559354    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Aug 04 03:14:33.560164 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.559540    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f5a694l9d4thg\""
Aug 04 03:14:33.560164 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.559550    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Aug 04 03:14:33.560164 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.559616    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Aug 04 03:14:33.563297 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.563278    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Aug 04 03:14:33.563402 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.563382    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Aug 04 03:14:33.566152 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.566133    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Aug 04 03:14:33.568751 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.568723    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Aug 04 03:14:33.571060 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.571037    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 03:14:33.642665 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.642627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.642665 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.642664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.642917 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.642763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.642917 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.642800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-web-config\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.642917 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.642834    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.642917 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.642860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-config\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.642917 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.642876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kfmn\" (UniqueName: \"kubernetes.io/projected/d89cee82-83f7-4b41-b3c5-344098aae10b-kube-api-access-9kfmn\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.642917 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.642895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d89cee82-83f7-4b41-b3c5-344098aae10b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.642917 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.642914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.643299 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.642939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.643299 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.643034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.643299 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.643066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.643299 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.643092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.643299 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.643136    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.643299 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.643167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.643299 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.643185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.643299 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.643204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.643299 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.643237    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.643299 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.643272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d89cee82-83f7-4b41-b3c5-344098aae10b-config-out\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.643787 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.643319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.743972 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.743934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.744426 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.744007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.744426 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.744034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.744426 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.744070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.744426 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.744092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.744426 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.744117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.744426 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.744143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.744426 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.744169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.744426 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.744212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d89cee82-83f7-4b41-b3c5-344098aae10b-config-out\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.744426 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.744237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.744426 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.744269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.744426 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.744291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.744426 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.744375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.745042 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.744774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.749111 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.745620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.749111 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.746326    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.749111 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.746924    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.749111 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.747285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.749111 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.747391    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.749111 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.747440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-web-config\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.749111 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.747475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.749111 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.747503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-config\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.749111 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.747527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kfmn\" (UniqueName: \"kubernetes.io/projected/d89cee82-83f7-4b41-b3c5-344098aae10b-kube-api-access-9kfmn\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.749111 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.747558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d89cee82-83f7-4b41-b3c5-344098aae10b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.749111 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.747588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.749111 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.747607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.749111 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.747620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.749111 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.747714    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.749111 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.748052    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.749111 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.748495    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.749111 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.748986    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.750013 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.748994    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.750106 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.750084    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-config\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.750874 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.750830    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.753246 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.753191    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d89cee82-83f7-4b41-b3c5-344098aae10b-config-out\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.753356 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.753286    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-web-config\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.753421 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.753369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d89cee82-83f7-4b41-b3c5-344098aae10b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.753421 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.753388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.753421 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.753396    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.756230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.756208    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kfmn\" (UniqueName: \"kubernetes.io/projected/d89cee82-83f7-4b41-b3c5-344098aae10b-kube-api-access-9kfmn\") pod \"prometheus-k8s-0\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:33.759767 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.759744    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9353a0a-4f31-42c2-bbbf-b623049e89c3","Type":"ContainerStarted","Data":"212b959a65044982323a26420dee76a62326ebd76e2dc6a7b41bde14dd4e86b5"}
Aug 04 03:14:33.759848 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.759810    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9353a0a-4f31-42c2-bbbf-b623049e89c3","Type":"ContainerStarted","Data":"516a47a0ebaa077d68c086ba50f56611fb27c0525e33553891e49ff1d51b4cdc"}
Aug 04 03:14:33.869954 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:33.869875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:34.033851 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:34.033711    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 03:14:34.036327 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:14:34.036302    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89cee82_83f7_4b41_b3c5_344098aae10b.slice/crio-7230e7af34a6c3ad0eb527994ee52f450799e6b4637b5329f9175619ab4fbfdf WatchSource:0}: Error finding container 7230e7af34a6c3ad0eb527994ee52f450799e6b4637b5329f9175619ab4fbfdf: Status 404 returned error can't find the container with id 7230e7af34a6c3ad0eb527994ee52f450799e6b4637b5329f9175619ab4fbfdf
Aug 04 03:14:34.765318 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:34.765279    2579 generic.go:356] "Generic (PLEG): container finished" podID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerID="56df7d99cacd4e5c7cca315a87373e469e2fd722db5e18bf59941edd85235cc6" exitCode=0
Aug 04 03:14:34.765775 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:34.765369    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d89cee82-83f7-4b41-b3c5-344098aae10b","Type":"ContainerDied","Data":"56df7d99cacd4e5c7cca315a87373e469e2fd722db5e18bf59941edd85235cc6"}
Aug 04 03:14:34.765775 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:34.765452    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d89cee82-83f7-4b41-b3c5-344098aae10b","Type":"ContainerStarted","Data":"7230e7af34a6c3ad0eb527994ee52f450799e6b4637b5329f9175619ab4fbfdf"}
Aug 04 03:14:34.768741 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:34.768720    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9353a0a-4f31-42c2-bbbf-b623049e89c3","Type":"ContainerStarted","Data":"1e92043fd3d56ec2a2c8d229ba6db16fc45cfea1cbebb7f5a0044e62a587d2f9"}
Aug 04 03:14:34.816562 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:34.816512    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.305862192 podStartE2EDuration="6.816497357s" totalImagesPullingTime="4.488383495s" totalInitContainerRuntime="22.25167ms" isStatefulPod=false podCreationTimestamp="2026-08-04 03:14:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:34.815126613 +0000 UTC m=+305.494913759" watchObservedRunningTime="2026-08-04 03:14:34.816497357 +0000 UTC m=+305.496284475"
Aug 04 03:14:34.838709 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:34.838684    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:14:35.234324 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:35.234288    2579 patch_prober.go:28] interesting pod/image-registry-958dbfdbb-wz2mz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Aug 04 03:14:35.234489 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:35.234346    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz" podUID="5f607efb-3bb4-4755-a366-4d8b0c5aa431" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 03:14:36.652771 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:36.652728    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:14:38.792044 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:38.792006    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d89cee82-83f7-4b41-b3c5-344098aae10b","Type":"ContainerStarted","Data":"32e38d6f72fb89b6f49e4c491f5b71795b955286a9ec2b580981fc55378f09c9"}
Aug 04 03:14:38.792483 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:38.792081    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d89cee82-83f7-4b41-b3c5-344098aae10b","Type":"ContainerStarted","Data":"156afea543258758d047ba014a700221788f1e6416ab83f706fcad1c113abe5e"}
Aug 04 03:14:39.877502 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:39.877473    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7966db58bc-xzdpq"]
Aug 04 03:14:39.909043 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:39.909018    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-569999c877-gdfxj"]
Aug 04 03:14:39.913419 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:39.913395    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:39.921332 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:39.921309    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-569999c877-gdfxj"]
Aug 04 03:14:40.032213 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.032183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-console-oauth-config\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.032365 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.032219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-console-config\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.032365 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.032262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-console-serving-cert\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.032365 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.032281    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-oauth-serving-cert\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.032581 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.032388    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-service-ca\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.032581 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.032452    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhrgt\" (UniqueName: \"kubernetes.io/projected/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-kube-api-access-nhrgt\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.032581 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.032526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-trusted-ca-bundle\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.133525 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.133483    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-console-serving-cert\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.133678 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.133530    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-oauth-serving-cert\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.133678 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.133572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-service-ca\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.133678 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.133607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhrgt\" (UniqueName: \"kubernetes.io/projected/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-kube-api-access-nhrgt\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.133678 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.133648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-trusted-ca-bundle\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.133880 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.133702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-console-oauth-config\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.133880 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.133736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-console-config\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.134716 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.134680    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-console-config\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.134716 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.134702    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-oauth-serving-cert\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.134849 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.134756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-service-ca\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.135072 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.135050    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-trusted-ca-bundle\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.136445 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.136090    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-console-serving-cert\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.136445 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.136403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-console-oauth-config\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.143497 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.143476    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhrgt\" (UniqueName: \"kubernetes.io/projected/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-kube-api-access-nhrgt\") pod \"console-569999c877-gdfxj\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") " pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.223682 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.223640    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:40.354883 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.354859    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-569999c877-gdfxj"]
Aug 04 03:14:40.357251 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:14:40.357230    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4afb1aa_1736_4e99_bedb_e28c5de8ab5e.slice/crio-c6a9393a7cbe2c259809d283635c9936eb41c67dffbedd817b81cb04b9aec442 WatchSource:0}: Error finding container c6a9393a7cbe2c259809d283635c9936eb41c67dffbedd817b81cb04b9aec442: Status 404 returned error can't find the container with id c6a9393a7cbe2c259809d283635c9936eb41c67dffbedd817b81cb04b9aec442
Aug 04 03:14:40.802987 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.802930    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-569999c877-gdfxj" event={"ID":"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e","Type":"ContainerStarted","Data":"93b0dd70bd6af7743098bf20379644e6a5bb923279127c8e32541e4a416e7fc8"}
Aug 04 03:14:40.803189 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.803003    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-569999c877-gdfxj" event={"ID":"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e","Type":"ContainerStarted","Data":"c6a9393a7cbe2c259809d283635c9936eb41c67dffbedd817b81cb04b9aec442"}
Aug 04 03:14:40.806174 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.806146    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d89cee82-83f7-4b41-b3c5-344098aae10b","Type":"ContainerStarted","Data":"05a2ba741498945fa241e05a32521f8ccf72431f63a45966e15674054df5f4b9"}
Aug 04 03:14:40.806270 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.806217    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d89cee82-83f7-4b41-b3c5-344098aae10b","Type":"ContainerStarted","Data":"41b46cbbf0d197cd5cd76d78740541337ee812440796d9e8756050c5fbc25c10"}
Aug 04 03:14:40.806270 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.806260    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d89cee82-83f7-4b41-b3c5-344098aae10b","Type":"ContainerStarted","Data":"558f003f867682700722e68851f0d7d8d668b77779a8474b1b1a9b1793cf92c5"}
Aug 04 03:14:40.806350 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.806299    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d89cee82-83f7-4b41-b3c5-344098aae10b","Type":"ContainerStarted","Data":"4de3833424d99a0dc7b57a474d484d4e9d4a89cc79e2bdc87b34ce0d008d9ddc"}
Aug 04 03:14:40.819642 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.819595    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-569999c877-gdfxj" podStartSLOduration=1.819580861 podStartE2EDuration="1.819580861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:40.819051589 +0000 UTC m=+311.498838728" watchObservedRunningTime="2026-08-04 03:14:40.819580861 +0000 UTC m=+311.499368044"
Aug 04 03:14:40.850307 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:40.850257    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.890648232 podStartE2EDuration="7.850239412s" totalImagesPullingTime="4.94117816s" totalInitContainerRuntime="18.41302ms" isStatefulPod=false podCreationTimestamp="2026-08-04 03:14:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:40.847353482 +0000 UTC m=+311.527140618" watchObservedRunningTime="2026-08-04 03:14:40.850239412 +0000 UTC m=+311.530026530"
Aug 04 03:14:41.782081 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:41.782044    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:14:42.175471 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:42.175376    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-958dbfdbb-wz2mz"]
Aug 04 03:14:43.870949 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:43.870921    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:14:50.224049 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:50.223987    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:50.224049 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:50.224059    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:50.228749 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:50.228728    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:50.852176 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:50.852144    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:14:50.894924 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:14:50.894888    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fb5db745b-p9pfz"]
Aug 04 03:15:04.897277 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:04.897207    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7966db58bc-xzdpq" podUID="ead8ec72-b5e9-4496-8e09-f32638fdadfd" containerName="console" containerID="cri-o://e8c1e65638f4263494c60ec5d3cfe624d087e339d4661a4c51f739fe4ebba3c7" gracePeriod=15
Aug 04 03:15:05.145245 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.145204    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:15:05.161471 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.161399    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-service-ca\") pod \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") "
Aug 04 03:15:05.161471 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.161436    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbhcs\" (UniqueName: \"kubernetes.io/projected/ead8ec72-b5e9-4496-8e09-f32638fdadfd-kube-api-access-cbhcs\") pod \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") "
Aug 04 03:15:05.161471 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.161462    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ead8ec72-b5e9-4496-8e09-f32638fdadfd-console-serving-cert\") pod \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") "
Aug 04 03:15:05.161688 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.161493    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-trusted-ca-bundle\") pod \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") "
Aug 04 03:15:05.161688 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.161535    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-console-config\") pod \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") "
Aug 04 03:15:05.161688 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.161576    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-oauth-serving-cert\") pod \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") "
Aug 04 03:15:05.161688 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.161602    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ead8ec72-b5e9-4496-8e09-f32638fdadfd-console-oauth-config\") pod \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\" (UID: \"ead8ec72-b5e9-4496-8e09-f32638fdadfd\") "
Aug 04 03:15:05.161985 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.161936    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-service-ca" (OuterVolumeSpecName: "service-ca") pod "ead8ec72-b5e9-4496-8e09-f32638fdadfd" (UID: "ead8ec72-b5e9-4496-8e09-f32638fdadfd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:15:05.162080 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.161945    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ead8ec72-b5e9-4496-8e09-f32638fdadfd" (UID: "ead8ec72-b5e9-4496-8e09-f32638fdadfd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:15:05.162080 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.162007    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ead8ec72-b5e9-4496-8e09-f32638fdadfd" (UID: "ead8ec72-b5e9-4496-8e09-f32638fdadfd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:15:05.162080 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.162023    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-console-config" (OuterVolumeSpecName: "console-config") pod "ead8ec72-b5e9-4496-8e09-f32638fdadfd" (UID: "ead8ec72-b5e9-4496-8e09-f32638fdadfd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:15:05.164233 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.164199    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead8ec72-b5e9-4496-8e09-f32638fdadfd-kube-api-access-cbhcs" (OuterVolumeSpecName: "kube-api-access-cbhcs") pod "ead8ec72-b5e9-4496-8e09-f32638fdadfd" (UID: "ead8ec72-b5e9-4496-8e09-f32638fdadfd"). InnerVolumeSpecName "kube-api-access-cbhcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:15:05.164353 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.164264    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ead8ec72-b5e9-4496-8e09-f32638fdadfd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ead8ec72-b5e9-4496-8e09-f32638fdadfd" (UID: "ead8ec72-b5e9-4496-8e09-f32638fdadfd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:05.164412 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.164356    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ead8ec72-b5e9-4496-8e09-f32638fdadfd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ead8ec72-b5e9-4496-8e09-f32638fdadfd" (UID: "ead8ec72-b5e9-4496-8e09-f32638fdadfd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:05.262414 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.262367    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:05.262414 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.262406    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbhcs\" (UniqueName: \"kubernetes.io/projected/ead8ec72-b5e9-4496-8e09-f32638fdadfd-kube-api-access-cbhcs\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:05.262414 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.262416    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ead8ec72-b5e9-4496-8e09-f32638fdadfd-console-serving-cert\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:05.262414 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.262426    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-trusted-ca-bundle\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:05.262684 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.262435    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-console-config\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:05.262684 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.262443    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ead8ec72-b5e9-4496-8e09-f32638fdadfd-oauth-serving-cert\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:05.262684 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.262453    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ead8ec72-b5e9-4496-8e09-f32638fdadfd-console-oauth-config\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:05.910193 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.910160    2579 generic.go:356] "Generic (PLEG): container finished" podID="ead8ec72-b5e9-4496-8e09-f32638fdadfd" containerID="e8c1e65638f4263494c60ec5d3cfe624d087e339d4661a4c51f739fe4ebba3c7" exitCode=0
Aug 04 03:15:05.910640 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.910203    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7966db58bc-xzdpq"
Aug 04 03:15:05.910640 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.910240    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7966db58bc-xzdpq" event={"ID":"ead8ec72-b5e9-4496-8e09-f32638fdadfd","Type":"ContainerDied","Data":"e8c1e65638f4263494c60ec5d3cfe624d087e339d4661a4c51f739fe4ebba3c7"}
Aug 04 03:15:05.910640 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.910286    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7966db58bc-xzdpq" event={"ID":"ead8ec72-b5e9-4496-8e09-f32638fdadfd","Type":"ContainerDied","Data":"00742dfa79b09a690d4bb63fc417ed5cf10a998a927bfcb1a4f0c34288589663"}
Aug 04 03:15:05.910640 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.910305    2579 scope.go:117] "RemoveContainer" containerID="e8c1e65638f4263494c60ec5d3cfe624d087e339d4661a4c51f739fe4ebba3c7"
Aug 04 03:15:05.918879 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.918862    2579 scope.go:117] "RemoveContainer" containerID="e8c1e65638f4263494c60ec5d3cfe624d087e339d4661a4c51f739fe4ebba3c7"
Aug 04 03:15:05.919153 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:15:05.919133    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c1e65638f4263494c60ec5d3cfe624d087e339d4661a4c51f739fe4ebba3c7\": container with ID starting with e8c1e65638f4263494c60ec5d3cfe624d087e339d4661a4c51f739fe4ebba3c7 not found: ID does not exist" containerID="e8c1e65638f4263494c60ec5d3cfe624d087e339d4661a4c51f739fe4ebba3c7"
Aug 04 03:15:05.919207 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.919162    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c1e65638f4263494c60ec5d3cfe624d087e339d4661a4c51f739fe4ebba3c7"} err="failed to get container status \"e8c1e65638f4263494c60ec5d3cfe624d087e339d4661a4c51f739fe4ebba3c7\": rpc error: code = NotFound desc = could not find container \"e8c1e65638f4263494c60ec5d3cfe624d087e339d4661a4c51f739fe4ebba3c7\": container with ID starting with e8c1e65638f4263494c60ec5d3cfe624d087e339d4661a4c51f739fe4ebba3c7 not found: ID does not exist"
Aug 04 03:15:05.925403 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.925381    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7966db58bc-xzdpq"]
Aug 04 03:15:05.929078 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:05.929047    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7966db58bc-xzdpq"]
Aug 04 03:15:07.194977 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.194908    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz" podUID="5f607efb-3bb4-4755-a366-4d8b0c5aa431" containerName="registry" containerID="cri-o://604abb67b072cde6693e803e1203aab19013f4be7a2529c8fcd99be0a5f90ab9" gracePeriod=30
Aug 04 03:15:07.439436 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.439402    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:15:07.481150 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.481112    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-certificates\") pod \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") "
Aug 04 03:15:07.481321 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.481156    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f607efb-3bb4-4755-a366-4d8b0c5aa431-trusted-ca\") pod \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") "
Aug 04 03:15:07.481321 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.481199    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f607efb-3bb4-4755-a366-4d8b0c5aa431-installation-pull-secrets\") pod \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") "
Aug 04 03:15:07.481396 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.481377    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-bound-sa-token\") pod \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") "
Aug 04 03:15:07.481465 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.481448    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f607efb-3bb4-4755-a366-4d8b0c5aa431-image-registry-private-configuration\") pod \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") "
Aug 04 03:15:07.481518 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.481501    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f607efb-3bb4-4755-a366-4d8b0c5aa431-ca-trust-extracted\") pod \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") "
Aug 04 03:15:07.481575 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.481540    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls\") pod \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") "
Aug 04 03:15:07.481627 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.481568    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5f607efb-3bb4-4755-a366-4d8b0c5aa431" (UID: "5f607efb-3bb4-4755-a366-4d8b0c5aa431"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:15:07.481627 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.481581    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz6tw\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-kube-api-access-zz6tw\") pod \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\" (UID: \"5f607efb-3bb4-4755-a366-4d8b0c5aa431\") "
Aug 04 03:15:07.481627 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.481603    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f607efb-3bb4-4755-a366-4d8b0c5aa431-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5f607efb-3bb4-4755-a366-4d8b0c5aa431" (UID: "5f607efb-3bb4-4755-a366-4d8b0c5aa431"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:15:07.481900 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.481875    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-certificates\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:07.481900 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.481901    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f607efb-3bb4-4755-a366-4d8b0c5aa431-trusted-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:07.483888 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.483855    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5f607efb-3bb4-4755-a366-4d8b0c5aa431" (UID: "5f607efb-3bb4-4755-a366-4d8b0c5aa431"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:15:07.484093 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.483876    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f607efb-3bb4-4755-a366-4d8b0c5aa431-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5f607efb-3bb4-4755-a366-4d8b0c5aa431" (UID: "5f607efb-3bb4-4755-a366-4d8b0c5aa431"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:07.484093 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.483976    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f607efb-3bb4-4755-a366-4d8b0c5aa431-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5f607efb-3bb4-4755-a366-4d8b0c5aa431" (UID: "5f607efb-3bb4-4755-a366-4d8b0c5aa431"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:07.484228 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.484195    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-kube-api-access-zz6tw" (OuterVolumeSpecName: "kube-api-access-zz6tw") pod "5f607efb-3bb4-4755-a366-4d8b0c5aa431" (UID: "5f607efb-3bb4-4755-a366-4d8b0c5aa431"). InnerVolumeSpecName "kube-api-access-zz6tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:15:07.484280 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.484241    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5f607efb-3bb4-4755-a366-4d8b0c5aa431" (UID: "5f607efb-3bb4-4755-a366-4d8b0c5aa431"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:15:07.489755 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.489729    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f607efb-3bb4-4755-a366-4d8b0c5aa431-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5f607efb-3bb4-4755-a366-4d8b0c5aa431" (UID: "5f607efb-3bb4-4755-a366-4d8b0c5aa431"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 03:15:07.582984 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.582917    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f607efb-3bb4-4755-a366-4d8b0c5aa431-installation-pull-secrets\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:07.582984 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.582951    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-bound-sa-token\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:07.582984 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.582986    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f607efb-3bb4-4755-a366-4d8b0c5aa431-image-registry-private-configuration\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:07.582984 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.582996    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f607efb-3bb4-4755-a366-4d8b0c5aa431-ca-trust-extracted\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:07.583241 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.583005    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-registry-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:07.583241 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.583013    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz6tw\" (UniqueName: \"kubernetes.io/projected/5f607efb-3bb4-4755-a366-4d8b0c5aa431-kube-api-access-zz6tw\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:07.844654 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.844575    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ead8ec72-b5e9-4496-8e09-f32638fdadfd" path="/var/lib/kubelet/pods/ead8ec72-b5e9-4496-8e09-f32638fdadfd/volumes"
Aug 04 03:15:07.927511 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.927482    2579 generic.go:356] "Generic (PLEG): container finished" podID="5f607efb-3bb4-4755-a366-4d8b0c5aa431" containerID="604abb67b072cde6693e803e1203aab19013f4be7a2529c8fcd99be0a5f90ab9" exitCode=0
Aug 04 03:15:07.927677 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.927530    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz"
Aug 04 03:15:07.927677 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.927570    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz" event={"ID":"5f607efb-3bb4-4755-a366-4d8b0c5aa431","Type":"ContainerDied","Data":"604abb67b072cde6693e803e1203aab19013f4be7a2529c8fcd99be0a5f90ab9"}
Aug 04 03:15:07.927677 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.927624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-958dbfdbb-wz2mz" event={"ID":"5f607efb-3bb4-4755-a366-4d8b0c5aa431","Type":"ContainerDied","Data":"607986b7ec77901dcf2b8eb59b54c6ac15f7c8343520a214e51bf431cf233d3b"}
Aug 04 03:15:07.927677 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.927652    2579 scope.go:117] "RemoveContainer" containerID="604abb67b072cde6693e803e1203aab19013f4be7a2529c8fcd99be0a5f90ab9"
Aug 04 03:15:07.936021 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.935995    2579 scope.go:117] "RemoveContainer" containerID="604abb67b072cde6693e803e1203aab19013f4be7a2529c8fcd99be0a5f90ab9"
Aug 04 03:15:07.936289 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:15:07.936269    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"604abb67b072cde6693e803e1203aab19013f4be7a2529c8fcd99be0a5f90ab9\": container with ID starting with 604abb67b072cde6693e803e1203aab19013f4be7a2529c8fcd99be0a5f90ab9 not found: ID does not exist" containerID="604abb67b072cde6693e803e1203aab19013f4be7a2529c8fcd99be0a5f90ab9"
Aug 04 03:15:07.936356 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.936303    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"604abb67b072cde6693e803e1203aab19013f4be7a2529c8fcd99be0a5f90ab9"} err="failed to get container status \"604abb67b072cde6693e803e1203aab19013f4be7a2529c8fcd99be0a5f90ab9\": rpc error: code = NotFound desc = could not find container \"604abb67b072cde6693e803e1203aab19013f4be7a2529c8fcd99be0a5f90ab9\": container with ID starting with 604abb67b072cde6693e803e1203aab19013f4be7a2529c8fcd99be0a5f90ab9 not found: ID does not exist"
Aug 04 03:15:07.943624 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.943594    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-958dbfdbb-wz2mz"]
Aug 04 03:15:07.949260 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:07.949236    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-958dbfdbb-wz2mz"]
Aug 04 03:15:09.844006 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:09.843978    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f607efb-3bb4-4755-a366-4d8b0c5aa431" path="/var/lib/kubelet/pods/5f607efb-3bb4-4755-a366-4d8b0c5aa431/volumes"
Aug 04 03:15:15.914352 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:15.914305    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5fb5db745b-p9pfz" podUID="3782a733-7495-46a9-bb37-acf3df0fb88f" containerName="console" containerID="cri-o://3b51df457481f04585e8be97e2d20e1d35a968f48393b6a4a43a02cfd15b71ca" gracePeriod=15
Aug 04 03:15:16.163594 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.163571    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:15:16.254511 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.254473    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3782a733-7495-46a9-bb37-acf3df0fb88f-oauth-serving-cert\") pod \"3782a733-7495-46a9-bb37-acf3df0fb88f\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") "
Aug 04 03:15:16.254692 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.254551    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3782a733-7495-46a9-bb37-acf3df0fb88f-console-serving-cert\") pod \"3782a733-7495-46a9-bb37-acf3df0fb88f\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") "
Aug 04 03:15:16.254764 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.254700    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3782a733-7495-46a9-bb37-acf3df0fb88f-service-ca\") pod \"3782a733-7495-46a9-bb37-acf3df0fb88f\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") "
Aug 04 03:15:16.254764 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.254752    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfjd5\" (UniqueName: \"kubernetes.io/projected/3782a733-7495-46a9-bb37-acf3df0fb88f-kube-api-access-cfjd5\") pod \"3782a733-7495-46a9-bb37-acf3df0fb88f\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") "
Aug 04 03:15:16.254878 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.254831    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3782a733-7495-46a9-bb37-acf3df0fb88f-console-oauth-config\") pod \"3782a733-7495-46a9-bb37-acf3df0fb88f\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") "
Aug 04 03:15:16.254878 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.254870    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3782a733-7495-46a9-bb37-acf3df0fb88f-console-config\") pod \"3782a733-7495-46a9-bb37-acf3df0fb88f\" (UID: \"3782a733-7495-46a9-bb37-acf3df0fb88f\") "
Aug 04 03:15:16.254996 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.254916    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3782a733-7495-46a9-bb37-acf3df0fb88f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3782a733-7495-46a9-bb37-acf3df0fb88f" (UID: "3782a733-7495-46a9-bb37-acf3df0fb88f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:15:16.255361 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.255171    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3782a733-7495-46a9-bb37-acf3df0fb88f-oauth-serving-cert\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:16.255361 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.255306    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3782a733-7495-46a9-bb37-acf3df0fb88f-service-ca" (OuterVolumeSpecName: "service-ca") pod "3782a733-7495-46a9-bb37-acf3df0fb88f" (UID: "3782a733-7495-46a9-bb37-acf3df0fb88f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:15:16.255690 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.255651    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3782a733-7495-46a9-bb37-acf3df0fb88f-console-config" (OuterVolumeSpecName: "console-config") pod "3782a733-7495-46a9-bb37-acf3df0fb88f" (UID: "3782a733-7495-46a9-bb37-acf3df0fb88f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:15:16.257316 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.257290    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3782a733-7495-46a9-bb37-acf3df0fb88f-kube-api-access-cfjd5" (OuterVolumeSpecName: "kube-api-access-cfjd5") pod "3782a733-7495-46a9-bb37-acf3df0fb88f" (UID: "3782a733-7495-46a9-bb37-acf3df0fb88f"). InnerVolumeSpecName "kube-api-access-cfjd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:15:16.257443 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.257370    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3782a733-7495-46a9-bb37-acf3df0fb88f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3782a733-7495-46a9-bb37-acf3df0fb88f" (UID: "3782a733-7495-46a9-bb37-acf3df0fb88f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:16.257631 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.257607    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3782a733-7495-46a9-bb37-acf3df0fb88f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3782a733-7495-46a9-bb37-acf3df0fb88f" (UID: "3782a733-7495-46a9-bb37-acf3df0fb88f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:16.356279 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.356249    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3782a733-7495-46a9-bb37-acf3df0fb88f-console-oauth-config\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:16.356279 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.356275    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3782a733-7495-46a9-bb37-acf3df0fb88f-console-config\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:16.356279 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.356285    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3782a733-7495-46a9-bb37-acf3df0fb88f-console-serving-cert\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:16.356499 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.356294    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3782a733-7495-46a9-bb37-acf3df0fb88f-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:16.356499 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.356303    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfjd5\" (UniqueName: \"kubernetes.io/projected/3782a733-7495-46a9-bb37-acf3df0fb88f-kube-api-access-cfjd5\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:16.964729 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.964696    2579 generic.go:356] "Generic (PLEG): container finished" podID="3782a733-7495-46a9-bb37-acf3df0fb88f" containerID="3b51df457481f04585e8be97e2d20e1d35a968f48393b6a4a43a02cfd15b71ca" exitCode=0
Aug 04 03:15:16.965157 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.964745    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fb5db745b-p9pfz" event={"ID":"3782a733-7495-46a9-bb37-acf3df0fb88f","Type":"ContainerDied","Data":"3b51df457481f04585e8be97e2d20e1d35a968f48393b6a4a43a02cfd15b71ca"}
Aug 04 03:15:16.965157 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.964760    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fb5db745b-p9pfz"
Aug 04 03:15:16.965157 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.964786    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fb5db745b-p9pfz" event={"ID":"3782a733-7495-46a9-bb37-acf3df0fb88f","Type":"ContainerDied","Data":"fd6a9af02fd04b7b18e0abb83f76141b336aa5e616c4b131d8c5225e33e8aa0a"}
Aug 04 03:15:16.965157 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.964805    2579 scope.go:117] "RemoveContainer" containerID="3b51df457481f04585e8be97e2d20e1d35a968f48393b6a4a43a02cfd15b71ca"
Aug 04 03:15:16.975148 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.975054    2579 scope.go:117] "RemoveContainer" containerID="3b51df457481f04585e8be97e2d20e1d35a968f48393b6a4a43a02cfd15b71ca"
Aug 04 03:15:16.975451 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:15:16.975420    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b51df457481f04585e8be97e2d20e1d35a968f48393b6a4a43a02cfd15b71ca\": container with ID starting with 3b51df457481f04585e8be97e2d20e1d35a968f48393b6a4a43a02cfd15b71ca not found: ID does not exist" containerID="3b51df457481f04585e8be97e2d20e1d35a968f48393b6a4a43a02cfd15b71ca"
Aug 04 03:15:16.975577 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.975462    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b51df457481f04585e8be97e2d20e1d35a968f48393b6a4a43a02cfd15b71ca"} err="failed to get container status \"3b51df457481f04585e8be97e2d20e1d35a968f48393b6a4a43a02cfd15b71ca\": rpc error: code = NotFound desc = could not find container \"3b51df457481f04585e8be97e2d20e1d35a968f48393b6a4a43a02cfd15b71ca\": container with ID starting with 3b51df457481f04585e8be97e2d20e1d35a968f48393b6a4a43a02cfd15b71ca not found: ID does not exist"
Aug 04 03:15:16.987886 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.987863    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fb5db745b-p9pfz"]
Aug 04 03:15:16.993549 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:16.993528    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fb5db745b-p9pfz"]
Aug 04 03:15:17.845975 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:17.845926    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3782a733-7495-46a9-bb37-acf3df0fb88f" path="/var/lib/kubelet/pods/3782a733-7495-46a9-bb37-acf3df0fb88f/volumes"
Aug 04 03:15:19.985063 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:19.985030    2579 generic.go:356] "Generic (PLEG): container finished" podID="6b243e88-7cd6-4eae-ad2e-399c8b43d676" containerID="ba3eecd4748cdaef184435139e69ef2cc77345e6cc091cd5bb2e674cfd7affbc" exitCode=0
Aug 04 03:15:19.985462 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:19.985102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6kmw8" event={"ID":"6b243e88-7cd6-4eae-ad2e-399c8b43d676","Type":"ContainerDied","Data":"ba3eecd4748cdaef184435139e69ef2cc77345e6cc091cd5bb2e674cfd7affbc"}
Aug 04 03:15:19.985516 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:19.985462    2579 scope.go:117] "RemoveContainer" containerID="ba3eecd4748cdaef184435139e69ef2cc77345e6cc091cd5bb2e674cfd7affbc"
Aug 04 03:15:20.991241 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:20.991209    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6kmw8" event={"ID":"6b243e88-7cd6-4eae-ad2e-399c8b43d676","Type":"ContainerStarted","Data":"b23346602d637ad3727a30b93ec36a959b3af629ea15080182dc0fe09a0fcafb"}
Aug 04 03:15:33.871028 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:33.870993    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:33.890535 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:33.890513    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:34.057509 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:34.057481    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:43.692223 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:43.692182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs\") pod \"network-metrics-daemon-c5d74\" (UID: \"b39560be-b9af-4ff7-8a5b-7882c66a03d8\") " pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:15:43.694624 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:43.694598    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b39560be-b9af-4ff7-8a5b-7882c66a03d8-metrics-certs\") pod \"network-metrics-daemon-c5d74\" (UID: \"b39560be-b9af-4ff7-8a5b-7882c66a03d8\") " pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:15:43.841629 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:43.841597    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6l47c\""
Aug 04 03:15:43.849925 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:43.849904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5d74"
Aug 04 03:15:43.978097 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:43.978075    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c5d74"]
Aug 04 03:15:43.980844 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:15:43.980805    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39560be_b9af_4ff7_8a5b_7882c66a03d8.slice/crio-ae381c1c7cf7126eb040735745cc4183620f594c0fb7986db36e65a5aa0f491a WatchSource:0}: Error finding container ae381c1c7cf7126eb040735745cc4183620f594c0fb7986db36e65a5aa0f491a: Status 404 returned error can't find the container with id ae381c1c7cf7126eb040735745cc4183620f594c0fb7986db36e65a5aa0f491a
Aug 04 03:15:44.082403 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:44.082370    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c5d74" event={"ID":"b39560be-b9af-4ff7-8a5b-7882c66a03d8","Type":"ContainerStarted","Data":"ae381c1c7cf7126eb040735745cc4183620f594c0fb7986db36e65a5aa0f491a"}
Aug 04 03:15:45.089226 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:45.089188    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c5d74" event={"ID":"b39560be-b9af-4ff7-8a5b-7882c66a03d8","Type":"ContainerStarted","Data":"243075aabb59d547d716901715d05e58ce11b9e12f8562094c8df1361e6ceceb"}
Aug 04 03:15:46.097524 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:46.097485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c5d74" event={"ID":"b39560be-b9af-4ff7-8a5b-7882c66a03d8","Type":"ContainerStarted","Data":"c744afbd26cf689373e69c55dca653088f6a51ad92febca5167301c8c2db898a"}
Aug 04 03:15:46.113364 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:46.113317    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-c5d74" podStartSLOduration=375.137909992 podStartE2EDuration="6m16.113302539s" totalImagesPullingTime="975.392547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:15:46.112842542 +0000 UTC m=+376.792629664" watchObservedRunningTime="2026-08-04 03:15:46.113302539 +0000 UTC m=+376.793089658"
Aug 04 03:15:47.654935 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:47.654903    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 03:15:47.655482 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:47.655195    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="alertmanager" containerID="cri-o://afea8032f4219653c0899bc3305723935ca338ea8d2d6a2ecf8abe996831c800" gracePeriod=120
Aug 04 03:15:47.655482 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:47.655250    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="kube-rbac-proxy-metric" containerID="cri-o://212b959a65044982323a26420dee76a62326ebd76e2dc6a7b41bde14dd4e86b5" gracePeriod=120
Aug 04 03:15:47.655482 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:47.655291    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="kube-rbac-proxy-web" containerID="cri-o://646d37c280d88d1c46b031ca16167c7a70671e0aff3472eb2ef046c53fb542f9" gracePeriod=120
Aug 04 03:15:47.655482 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:47.655306    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="prom-label-proxy" containerID="cri-o://1e92043fd3d56ec2a2c8d229ba6db16fc45cfea1cbebb7f5a0044e62a587d2f9" gracePeriod=120
Aug 04 03:15:47.655482 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:47.655342    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="kube-rbac-proxy" containerID="cri-o://516a47a0ebaa077d68c086ba50f56611fb27c0525e33553891e49ff1d51b4cdc" gracePeriod=120
Aug 04 03:15:47.655482 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:47.655352    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="config-reloader" containerID="cri-o://93dc098471b03a648ae9883e70586d0c801343c3e2a0ed5950048057f5e9b095" gracePeriod=120
Aug 04 03:15:48.109751 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.109719    2579 generic.go:356] "Generic (PLEG): container finished" podID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerID="1e92043fd3d56ec2a2c8d229ba6db16fc45cfea1cbebb7f5a0044e62a587d2f9" exitCode=0
Aug 04 03:15:48.109751 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.109742    2579 generic.go:356] "Generic (PLEG): container finished" podID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerID="516a47a0ebaa077d68c086ba50f56611fb27c0525e33553891e49ff1d51b4cdc" exitCode=0
Aug 04 03:15:48.109751 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.109749    2579 generic.go:356] "Generic (PLEG): container finished" podID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerID="93dc098471b03a648ae9883e70586d0c801343c3e2a0ed5950048057f5e9b095" exitCode=0
Aug 04 03:15:48.109751 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.109755    2579 generic.go:356] "Generic (PLEG): container finished" podID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerID="afea8032f4219653c0899bc3305723935ca338ea8d2d6a2ecf8abe996831c800" exitCode=0
Aug 04 03:15:48.110099 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.109795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9353a0a-4f31-42c2-bbbf-b623049e89c3","Type":"ContainerDied","Data":"1e92043fd3d56ec2a2c8d229ba6db16fc45cfea1cbebb7f5a0044e62a587d2f9"}
Aug 04 03:15:48.110099 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.109876    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9353a0a-4f31-42c2-bbbf-b623049e89c3","Type":"ContainerDied","Data":"516a47a0ebaa077d68c086ba50f56611fb27c0525e33553891e49ff1d51b4cdc"}
Aug 04 03:15:48.110099 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.109915    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9353a0a-4f31-42c2-bbbf-b623049e89c3","Type":"ContainerDied","Data":"93dc098471b03a648ae9883e70586d0c801343c3e2a0ed5950048057f5e9b095"}
Aug 04 03:15:48.110099 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.109953    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9353a0a-4f31-42c2-bbbf-b623049e89c3","Type":"ContainerDied","Data":"afea8032f4219653c0899bc3305723935ca338ea8d2d6a2ecf8abe996831c800"}
Aug 04 03:15:48.900912 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.900891    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:48.940513 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.940487    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9353a0a-4f31-42c2-bbbf-b623049e89c3-metrics-client-ca\") pod \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") "
Aug 04 03:15:48.940684 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.940518    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-cluster-tls-config\") pod \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") "
Aug 04 03:15:48.940684 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.940542    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-config-volume\") pod \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") "
Aug 04 03:15:48.940684 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.940557    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9353a0a-4f31-42c2-bbbf-b623049e89c3-tls-assets\") pod \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") "
Aug 04 03:15:48.940684 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.940579    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-kube-rbac-proxy\") pod \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") "
Aug 04 03:15:48.940684 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.940610    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9353a0a-4f31-42c2-bbbf-b623049e89c3-alertmanager-main-db\") pod \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") "
Aug 04 03:15:48.940955 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.940877    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9353a0a-4f31-42c2-bbbf-b623049e89c3-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "f9353a0a-4f31-42c2-bbbf-b623049e89c3" (UID: "f9353a0a-4f31-42c2-bbbf-b623049e89c3"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 03:15:48.941036 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.941020    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") "
Aug 04 03:15:48.941093 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.941077    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccnnc\" (UniqueName: \"kubernetes.io/projected/f9353a0a-4f31-42c2-bbbf-b623049e89c3-kube-api-access-ccnnc\") pod \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") "
Aug 04 03:15:48.941146 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.941112    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9353a0a-4f31-42c2-bbbf-b623049e89c3-alertmanager-trusted-ca-bundle\") pod \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") "
Aug 04 03:15:48.941146 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.941140    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-main-tls\") pod \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") "
Aug 04 03:15:48.941245 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.941185    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-web-config\") pod \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") "
Aug 04 03:15:48.941245 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.941213    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-kube-rbac-proxy-web\") pod \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") "
Aug 04 03:15:48.941601 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.941478    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9353a0a-4f31-42c2-bbbf-b623049e89c3-alertmanager-main-db\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:48.941723 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.941684    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9353a0a-4f31-42c2-bbbf-b623049e89c3-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "f9353a0a-4f31-42c2-bbbf-b623049e89c3" (UID: "f9353a0a-4f31-42c2-bbbf-b623049e89c3"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:15:48.941989 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.941940    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9353a0a-4f31-42c2-bbbf-b623049e89c3-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "f9353a0a-4f31-42c2-bbbf-b623049e89c3" (UID: "f9353a0a-4f31-42c2-bbbf-b623049e89c3"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:15:48.944020 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.943998    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-config-volume" (OuterVolumeSpecName: "config-volume") pod "f9353a0a-4f31-42c2-bbbf-b623049e89c3" (UID: "f9353a0a-4f31-42c2-bbbf-b623049e89c3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:48.944457 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.944422    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9353a0a-4f31-42c2-bbbf-b623049e89c3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f9353a0a-4f31-42c2-bbbf-b623049e89c3" (UID: "f9353a0a-4f31-42c2-bbbf-b623049e89c3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:15:48.944543 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.944452    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "f9353a0a-4f31-42c2-bbbf-b623049e89c3" (UID: "f9353a0a-4f31-42c2-bbbf-b623049e89c3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:48.944601 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.944554    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9353a0a-4f31-42c2-bbbf-b623049e89c3-kube-api-access-ccnnc" (OuterVolumeSpecName: "kube-api-access-ccnnc") pod "f9353a0a-4f31-42c2-bbbf-b623049e89c3" (UID: "f9353a0a-4f31-42c2-bbbf-b623049e89c3"). InnerVolumeSpecName "kube-api-access-ccnnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:15:48.945277 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.945252    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "f9353a0a-4f31-42c2-bbbf-b623049e89c3" (UID: "f9353a0a-4f31-42c2-bbbf-b623049e89c3"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:48.946100 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.946070    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "f9353a0a-4f31-42c2-bbbf-b623049e89c3" (UID: "f9353a0a-4f31-42c2-bbbf-b623049e89c3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:48.946208 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.946132    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "f9353a0a-4f31-42c2-bbbf-b623049e89c3" (UID: "f9353a0a-4f31-42c2-bbbf-b623049e89c3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:48.951508 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.949696    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "f9353a0a-4f31-42c2-bbbf-b623049e89c3" (UID: "f9353a0a-4f31-42c2-bbbf-b623049e89c3"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:48.958024 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:48.958003    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-web-config" (OuterVolumeSpecName: "web-config") pod "f9353a0a-4f31-42c2-bbbf-b623049e89c3" (UID: "f9353a0a-4f31-42c2-bbbf-b623049e89c3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:49.042083 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.042002    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9353a0a-4f31-42c2-bbbf-b623049e89c3-config-out\") pod \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\" (UID: \"f9353a0a-4f31-42c2-bbbf-b623049e89c3\") "
Aug 04 03:15:49.042234 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.042204    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-main-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:49.042234 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.042221    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-web-config\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:49.042356 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.042235    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:49.042356 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.042250    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9353a0a-4f31-42c2-bbbf-b623049e89c3-metrics-client-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:49.042356 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.042263    2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-cluster-tls-config\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:49.042356 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.042276    2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:49.042356 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.042289    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9353a0a-4f31-42c2-bbbf-b623049e89c3-tls-assets\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:49.042356 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.042301    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:49.042356 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.042316    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9353a0a-4f31-42c2-bbbf-b623049e89c3-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:49.042356 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.042337    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccnnc\" (UniqueName: \"kubernetes.io/projected/f9353a0a-4f31-42c2-bbbf-b623049e89c3-kube-api-access-ccnnc\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:49.042356 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.042350    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9353a0a-4f31-42c2-bbbf-b623049e89c3-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:49.044037 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.044017    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9353a0a-4f31-42c2-bbbf-b623049e89c3-config-out" (OuterVolumeSpecName: "config-out") pod "f9353a0a-4f31-42c2-bbbf-b623049e89c3" (UID: "f9353a0a-4f31-42c2-bbbf-b623049e89c3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 03:15:49.116289 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.116255    2579 generic.go:356] "Generic (PLEG): container finished" podID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerID="212b959a65044982323a26420dee76a62326ebd76e2dc6a7b41bde14dd4e86b5" exitCode=0
Aug 04 03:15:49.116289 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.116278    2579 generic.go:356] "Generic (PLEG): container finished" podID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerID="646d37c280d88d1c46b031ca16167c7a70671e0aff3472eb2ef046c53fb542f9" exitCode=0
Aug 04 03:15:49.116472 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.116298    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9353a0a-4f31-42c2-bbbf-b623049e89c3","Type":"ContainerDied","Data":"212b959a65044982323a26420dee76a62326ebd76e2dc6a7b41bde14dd4e86b5"}
Aug 04 03:15:49.116472 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.116332    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.116472 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.116356    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9353a0a-4f31-42c2-bbbf-b623049e89c3","Type":"ContainerDied","Data":"646d37c280d88d1c46b031ca16167c7a70671e0aff3472eb2ef046c53fb542f9"}
Aug 04 03:15:49.116472 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.116384    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9353a0a-4f31-42c2-bbbf-b623049e89c3","Type":"ContainerDied","Data":"7fabbba4c3e885e9c402a6c98707886826859f3ed8af1ae7c37a3477c472df30"}
Aug 04 03:15:49.116472 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.116415    2579 scope.go:117] "RemoveContainer" containerID="1e92043fd3d56ec2a2c8d229ba6db16fc45cfea1cbebb7f5a0044e62a587d2f9"
Aug 04 03:15:49.124609 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.124589    2579 scope.go:117] "RemoveContainer" containerID="212b959a65044982323a26420dee76a62326ebd76e2dc6a7b41bde14dd4e86b5"
Aug 04 03:15:49.131500 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.131481    2579 scope.go:117] "RemoveContainer" containerID="516a47a0ebaa077d68c086ba50f56611fb27c0525e33553891e49ff1d51b4cdc"
Aug 04 03:15:49.138153 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.138134    2579 scope.go:117] "RemoveContainer" containerID="646d37c280d88d1c46b031ca16167c7a70671e0aff3472eb2ef046c53fb542f9"
Aug 04 03:15:49.141487 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.141466    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 03:15:49.143121 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.143104    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9353a0a-4f31-42c2-bbbf-b623049e89c3-config-out\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:49.145991 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.145975    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 03:15:49.146061 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.146005    2579 scope.go:117] "RemoveContainer" containerID="93dc098471b03a648ae9883e70586d0c801343c3e2a0ed5950048057f5e9b095"
Aug 04 03:15:49.152714 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.152697    2579 scope.go:117] "RemoveContainer" containerID="afea8032f4219653c0899bc3305723935ca338ea8d2d6a2ecf8abe996831c800"
Aug 04 03:15:49.159339 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.159323    2579 scope.go:117] "RemoveContainer" containerID="15e6294804cbc205fdac5a04c756c971218d4a76b18293f295f3b108bfd5582f"
Aug 04 03:15:49.166159 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.166143    2579 scope.go:117] "RemoveContainer" containerID="1e92043fd3d56ec2a2c8d229ba6db16fc45cfea1cbebb7f5a0044e62a587d2f9"
Aug 04 03:15:49.166396 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:15:49.166378    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e92043fd3d56ec2a2c8d229ba6db16fc45cfea1cbebb7f5a0044e62a587d2f9\": container with ID starting with 1e92043fd3d56ec2a2c8d229ba6db16fc45cfea1cbebb7f5a0044e62a587d2f9 not found: ID does not exist" containerID="1e92043fd3d56ec2a2c8d229ba6db16fc45cfea1cbebb7f5a0044e62a587d2f9"
Aug 04 03:15:49.166440 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.166403    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e92043fd3d56ec2a2c8d229ba6db16fc45cfea1cbebb7f5a0044e62a587d2f9"} err="failed to get container status \"1e92043fd3d56ec2a2c8d229ba6db16fc45cfea1cbebb7f5a0044e62a587d2f9\": rpc error: code = NotFound desc = could not find container \"1e92043fd3d56ec2a2c8d229ba6db16fc45cfea1cbebb7f5a0044e62a587d2f9\": container with ID starting with 1e92043fd3d56ec2a2c8d229ba6db16fc45cfea1cbebb7f5a0044e62a587d2f9 not found: ID does not exist"
Aug 04 03:15:49.166440 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.166420    2579 scope.go:117] "RemoveContainer" containerID="212b959a65044982323a26420dee76a62326ebd76e2dc6a7b41bde14dd4e86b5"
Aug 04 03:15:49.166671 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:15:49.166653    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212b959a65044982323a26420dee76a62326ebd76e2dc6a7b41bde14dd4e86b5\": container with ID starting with 212b959a65044982323a26420dee76a62326ebd76e2dc6a7b41bde14dd4e86b5 not found: ID does not exist" containerID="212b959a65044982323a26420dee76a62326ebd76e2dc6a7b41bde14dd4e86b5"
Aug 04 03:15:49.166720 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.166675    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212b959a65044982323a26420dee76a62326ebd76e2dc6a7b41bde14dd4e86b5"} err="failed to get container status \"212b959a65044982323a26420dee76a62326ebd76e2dc6a7b41bde14dd4e86b5\": rpc error: code = NotFound desc = could not find container \"212b959a65044982323a26420dee76a62326ebd76e2dc6a7b41bde14dd4e86b5\": container with ID starting with 212b959a65044982323a26420dee76a62326ebd76e2dc6a7b41bde14dd4e86b5 not found: ID does not exist"
Aug 04 03:15:49.166720 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.166688    2579 scope.go:117] "RemoveContainer" containerID="516a47a0ebaa077d68c086ba50f56611fb27c0525e33553891e49ff1d51b4cdc"
Aug 04 03:15:49.166883 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:15:49.166867    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516a47a0ebaa077d68c086ba50f56611fb27c0525e33553891e49ff1d51b4cdc\": container with ID starting with 516a47a0ebaa077d68c086ba50f56611fb27c0525e33553891e49ff1d51b4cdc not found: ID does not exist" containerID="516a47a0ebaa077d68c086ba50f56611fb27c0525e33553891e49ff1d51b4cdc"
Aug 04 03:15:49.166922 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.166886    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516a47a0ebaa077d68c086ba50f56611fb27c0525e33553891e49ff1d51b4cdc"} err="failed to get container status \"516a47a0ebaa077d68c086ba50f56611fb27c0525e33553891e49ff1d51b4cdc\": rpc error: code = NotFound desc = could not find container \"516a47a0ebaa077d68c086ba50f56611fb27c0525e33553891e49ff1d51b4cdc\": container with ID starting with 516a47a0ebaa077d68c086ba50f56611fb27c0525e33553891e49ff1d51b4cdc not found: ID does not exist"
Aug 04 03:15:49.166922 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.166899    2579 scope.go:117] "RemoveContainer" containerID="646d37c280d88d1c46b031ca16167c7a70671e0aff3472eb2ef046c53fb542f9"
Aug 04 03:15:49.167111 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:15:49.167096    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646d37c280d88d1c46b031ca16167c7a70671e0aff3472eb2ef046c53fb542f9\": container with ID starting with 646d37c280d88d1c46b031ca16167c7a70671e0aff3472eb2ef046c53fb542f9 not found: ID does not exist" containerID="646d37c280d88d1c46b031ca16167c7a70671e0aff3472eb2ef046c53fb542f9"
Aug 04 03:15:49.167153 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.167114    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646d37c280d88d1c46b031ca16167c7a70671e0aff3472eb2ef046c53fb542f9"} err="failed to get container status \"646d37c280d88d1c46b031ca16167c7a70671e0aff3472eb2ef046c53fb542f9\": rpc error: code = NotFound desc = could not find container \"646d37c280d88d1c46b031ca16167c7a70671e0aff3472eb2ef046c53fb542f9\": container with ID starting with 646d37c280d88d1c46b031ca16167c7a70671e0aff3472eb2ef046c53fb542f9 not found: ID does not exist"
Aug 04 03:15:49.167153 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.167126    2579 scope.go:117] "RemoveContainer" containerID="93dc098471b03a648ae9883e70586d0c801343c3e2a0ed5950048057f5e9b095"
Aug 04 03:15:49.167334 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:15:49.167319    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93dc098471b03a648ae9883e70586d0c801343c3e2a0ed5950048057f5e9b095\": container with ID starting with 93dc098471b03a648ae9883e70586d0c801343c3e2a0ed5950048057f5e9b095 not found: ID does not exist" containerID="93dc098471b03a648ae9883e70586d0c801343c3e2a0ed5950048057f5e9b095"
Aug 04 03:15:49.167420 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.167336    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93dc098471b03a648ae9883e70586d0c801343c3e2a0ed5950048057f5e9b095"} err="failed to get container status \"93dc098471b03a648ae9883e70586d0c801343c3e2a0ed5950048057f5e9b095\": rpc error: code = NotFound desc = could not find container \"93dc098471b03a648ae9883e70586d0c801343c3e2a0ed5950048057f5e9b095\": container with ID starting with 93dc098471b03a648ae9883e70586d0c801343c3e2a0ed5950048057f5e9b095 not found: ID does not exist"
Aug 04 03:15:49.167420 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.167347    2579 scope.go:117] "RemoveContainer" containerID="afea8032f4219653c0899bc3305723935ca338ea8d2d6a2ecf8abe996831c800"
Aug 04 03:15:49.167525 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:15:49.167512    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afea8032f4219653c0899bc3305723935ca338ea8d2d6a2ecf8abe996831c800\": container with ID starting with afea8032f4219653c0899bc3305723935ca338ea8d2d6a2ecf8abe996831c800 not found: ID does not exist" containerID="afea8032f4219653c0899bc3305723935ca338ea8d2d6a2ecf8abe996831c800"
Aug 04 03:15:49.167560 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.167529    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afea8032f4219653c0899bc3305723935ca338ea8d2d6a2ecf8abe996831c800"} err="failed to get container status \"afea8032f4219653c0899bc3305723935ca338ea8d2d6a2ecf8abe996831c800\": rpc error: code = NotFound desc = could not find container \"afea8032f4219653c0899bc3305723935ca338ea8d2d6a2ecf8abe996831c800\": container with ID starting with afea8032f4219653c0899bc3305723935ca338ea8d2d6a2ecf8abe996831c800 not found: ID does not exist"
Aug 04 03:15:49.167560 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.167540    2579 scope.go:117] "RemoveContainer" containerID="15e6294804cbc205fdac5a04c756c971218d4a76b18293f295f3b108bfd5582f"
Aug 04 03:15:49.167768 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:15:49.167721    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e6294804cbc205fdac5a04c756c971218d4a76b18293f295f3b108bfd5582f\": container with ID starting with 15e6294804cbc205fdac5a04c756c971218d4a76b18293f295f3b108bfd5582f not found: ID does not exist" containerID="15e6294804cbc205fdac5a04c756c971218d4a76b18293f295f3b108bfd5582f"
Aug 04 03:15:49.167825 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.167771    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e6294804cbc205fdac5a04c756c971218d4a76b18293f295f3b108bfd5582f"} err="failed to get container status \"15e6294804cbc205fdac5a04c756c971218d4a76b18293f295f3b108bfd5582f\": rpc error: code = NotFound desc = could not find container \"15e6294804cbc205fdac5a04c756c971218d4a76b18293f295f3b108bfd5582f\": container with ID starting with 15e6294804cbc205fdac5a04c756c971218d4a76b18293f295f3b108bfd5582f not found: ID does not exist"
Aug 04 03:15:49.167825 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.167783    2579 scope.go:117] "RemoveContainer" containerID="1e92043fd3d56ec2a2c8d229ba6db16fc45cfea1cbebb7f5a0044e62a587d2f9"
Aug 04 03:15:49.168611 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.168060    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e92043fd3d56ec2a2c8d229ba6db16fc45cfea1cbebb7f5a0044e62a587d2f9"} err="failed to get container status \"1e92043fd3d56ec2a2c8d229ba6db16fc45cfea1cbebb7f5a0044e62a587d2f9\": rpc error: code = NotFound desc = could not find container \"1e92043fd3d56ec2a2c8d229ba6db16fc45cfea1cbebb7f5a0044e62a587d2f9\": container with ID starting with 1e92043fd3d56ec2a2c8d229ba6db16fc45cfea1cbebb7f5a0044e62a587d2f9 not found: ID does not exist"
Aug 04 03:15:49.168714 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.168619    2579 scope.go:117] "RemoveContainer" containerID="212b959a65044982323a26420dee76a62326ebd76e2dc6a7b41bde14dd4e86b5"
Aug 04 03:15:49.168920 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.168894    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212b959a65044982323a26420dee76a62326ebd76e2dc6a7b41bde14dd4e86b5"} err="failed to get container status \"212b959a65044982323a26420dee76a62326ebd76e2dc6a7b41bde14dd4e86b5\": rpc error: code = NotFound desc = could not find container \"212b959a65044982323a26420dee76a62326ebd76e2dc6a7b41bde14dd4e86b5\": container with ID starting with 212b959a65044982323a26420dee76a62326ebd76e2dc6a7b41bde14dd4e86b5 not found: ID does not exist"
Aug 04 03:15:49.169012 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.168922    2579 scope.go:117] "RemoveContainer" containerID="516a47a0ebaa077d68c086ba50f56611fb27c0525e33553891e49ff1d51b4cdc"
Aug 04 03:15:49.169275 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.169254    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516a47a0ebaa077d68c086ba50f56611fb27c0525e33553891e49ff1d51b4cdc"} err="failed to get container status \"516a47a0ebaa077d68c086ba50f56611fb27c0525e33553891e49ff1d51b4cdc\": rpc error: code = NotFound desc = could not find container \"516a47a0ebaa077d68c086ba50f56611fb27c0525e33553891e49ff1d51b4cdc\": container with ID starting with 516a47a0ebaa077d68c086ba50f56611fb27c0525e33553891e49ff1d51b4cdc not found: ID does not exist"
Aug 04 03:15:49.169328 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.169276    2579 scope.go:117] "RemoveContainer" containerID="646d37c280d88d1c46b031ca16167c7a70671e0aff3472eb2ef046c53fb542f9"
Aug 04 03:15:49.169523 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.169507    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646d37c280d88d1c46b031ca16167c7a70671e0aff3472eb2ef046c53fb542f9"} err="failed to get container status \"646d37c280d88d1c46b031ca16167c7a70671e0aff3472eb2ef046c53fb542f9\": rpc error: code = NotFound desc = could not find container \"646d37c280d88d1c46b031ca16167c7a70671e0aff3472eb2ef046c53fb542f9\": container with ID starting with 646d37c280d88d1c46b031ca16167c7a70671e0aff3472eb2ef046c53fb542f9 not found: ID does not exist"
Aug 04 03:15:49.169571 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.169531    2579 scope.go:117] "RemoveContainer" containerID="93dc098471b03a648ae9883e70586d0c801343c3e2a0ed5950048057f5e9b095"
Aug 04 03:15:49.169763 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.169747    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93dc098471b03a648ae9883e70586d0c801343c3e2a0ed5950048057f5e9b095"} err="failed to get container status \"93dc098471b03a648ae9883e70586d0c801343c3e2a0ed5950048057f5e9b095\": rpc error: code = NotFound desc = could not find container \"93dc098471b03a648ae9883e70586d0c801343c3e2a0ed5950048057f5e9b095\": container with ID starting with 93dc098471b03a648ae9883e70586d0c801343c3e2a0ed5950048057f5e9b095 not found: ID does not exist"
Aug 04 03:15:49.169821 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.169770    2579 scope.go:117] "RemoveContainer" containerID="afea8032f4219653c0899bc3305723935ca338ea8d2d6a2ecf8abe996831c800"
Aug 04 03:15:49.170053 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.169952    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afea8032f4219653c0899bc3305723935ca338ea8d2d6a2ecf8abe996831c800"} err="failed to get container status \"afea8032f4219653c0899bc3305723935ca338ea8d2d6a2ecf8abe996831c800\": rpc error: code = NotFound desc = could not find container \"afea8032f4219653c0899bc3305723935ca338ea8d2d6a2ecf8abe996831c800\": container with ID starting with afea8032f4219653c0899bc3305723935ca338ea8d2d6a2ecf8abe996831c800 not found: ID does not exist"
Aug 04 03:15:49.170107 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170056    2579 scope.go:117] "RemoveContainer" containerID="15e6294804cbc205fdac5a04c756c971218d4a76b18293f295f3b108bfd5582f"
Aug 04 03:15:49.170254 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170239    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 03:15:49.170302 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170278    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e6294804cbc205fdac5a04c756c971218d4a76b18293f295f3b108bfd5582f"} err="failed to get container status \"15e6294804cbc205fdac5a04c756c971218d4a76b18293f295f3b108bfd5582f\": rpc error: code = NotFound desc = could not find container \"15e6294804cbc205fdac5a04c756c971218d4a76b18293f295f3b108bfd5582f\": container with ID starting with 15e6294804cbc205fdac5a04c756c971218d4a76b18293f295f3b108bfd5582f not found: ID does not exist"
Aug 04 03:15:49.170526 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170515    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="config-reloader"
Aug 04 03:15:49.170562 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170529    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="config-reloader"
Aug 04 03:15:49.170562 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170539    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="init-config-reloader"
Aug 04 03:15:49.170562 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170545    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="init-config-reloader"
Aug 04 03:15:49.170562 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170552    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="kube-rbac-proxy-metric"
Aug 04 03:15:49.170562 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170558    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="kube-rbac-proxy-metric"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170564    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="kube-rbac-proxy-web"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170569    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="kube-rbac-proxy-web"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170581    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="alertmanager"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170586    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="alertmanager"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170591    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="prom-label-proxy"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170596    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="prom-label-proxy"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170602    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3782a733-7495-46a9-bb37-acf3df0fb88f" containerName="console"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170607    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3782a733-7495-46a9-bb37-acf3df0fb88f" containerName="console"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170612    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="kube-rbac-proxy"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170616    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="kube-rbac-proxy"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170626    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f607efb-3bb4-4755-a366-4d8b0c5aa431" containerName="registry"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170630    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f607efb-3bb4-4755-a366-4d8b0c5aa431" containerName="registry"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170638    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ead8ec72-b5e9-4496-8e09-f32638fdadfd" containerName="console"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170643    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ead8ec72-b5e9-4496-8e09-f32638fdadfd" containerName="console"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170685    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3782a733-7495-46a9-bb37-acf3df0fb88f" containerName="console"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170691    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="kube-rbac-proxy"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170696    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="kube-rbac-proxy-metric"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170703    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ead8ec72-b5e9-4496-8e09-f32638fdadfd" containerName="console"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170709    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="config-reloader"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170714    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="kube-rbac-proxy-web"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170719    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="prom-label-proxy"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170726    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" containerName="alertmanager"
Aug 04 03:15:49.170859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.170732    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f607efb-3bb4-4755-a366-4d8b0c5aa431" containerName="registry"
Aug 04 03:15:49.177455 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.177431    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.180151 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.180133    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Aug 04 03:15:49.180300 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.180284    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Aug 04 03:15:49.180383 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.180302    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Aug 04 03:15:49.180436 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.180409    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Aug 04 03:15:49.180484 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.180467    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Aug 04 03:15:49.180707 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.180691    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pmm9s\""
Aug 04 03:15:49.180794 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.180720    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Aug 04 03:15:49.180794 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.180767    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Aug 04 03:15:49.180941 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.180927    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Aug 04 03:15:49.185199 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.185180    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Aug 04 03:15:49.190698 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.190679    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 03:15:49.243505 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.243477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.243646 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.243513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trd5k\" (UniqueName: \"kubernetes.io/projected/45c01697-5168-4b0e-ac9b-ca821db8c611-kube-api-access-trd5k\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.243646 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.243531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.243646 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.243548    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.243646 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.243602    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45c01697-5168-4b0e-ac9b-ca821db8c611-config-out\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.243646 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.243625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/45c01697-5168-4b0e-ac9b-ca821db8c611-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.243838 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.243646    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c01697-5168-4b0e-ac9b-ca821db8c611-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.243838 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.243674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-web-config\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.243838 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.243712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45c01697-5168-4b0e-ac9b-ca821db8c611-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.243838 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.243785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45c01697-5168-4b0e-ac9b-ca821db8c611-tls-assets\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.243838 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.243816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.244013 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.243842    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-config-volume\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.244013 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.243890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.344701 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.344636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.344701 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.344673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.344701 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.344696    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trd5k\" (UniqueName: \"kubernetes.io/projected/45c01697-5168-4b0e-ac9b-ca821db8c611-kube-api-access-trd5k\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.344864 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.344715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.344864 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.344733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.344864 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.344763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45c01697-5168-4b0e-ac9b-ca821db8c611-config-out\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.344864 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.344797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/45c01697-5168-4b0e-ac9b-ca821db8c611-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.344864 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.344832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c01697-5168-4b0e-ac9b-ca821db8c611-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.345094 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.344908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-web-config\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.345094 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.344929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45c01697-5168-4b0e-ac9b-ca821db8c611-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.345094 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.344955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45c01697-5168-4b0e-ac9b-ca821db8c611-tls-assets\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.345094 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.345009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.345094 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.345044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-config-volume\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.347205 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.345779    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45c01697-5168-4b0e-ac9b-ca821db8c611-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.347205 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.346614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c01697-5168-4b0e-ac9b-ca821db8c611-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.347205 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.347148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/45c01697-5168-4b0e-ac9b-ca821db8c611-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.347750 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.347727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45c01697-5168-4b0e-ac9b-ca821db8c611-config-out\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.347750 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.347741    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.347894 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.347825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.347951 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.347889    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.348039 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.348014    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.348039 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.348015    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45c01697-5168-4b0e-ac9b-ca821db8c611-tls-assets\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.348151 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.348087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-config-volume\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.349403 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.349379    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-web-config\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.349504 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.349489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/45c01697-5168-4b0e-ac9b-ca821db8c611-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.352325 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.352306    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trd5k\" (UniqueName: \"kubernetes.io/projected/45c01697-5168-4b0e-ac9b-ca821db8c611-kube-api-access-trd5k\") pod \"alertmanager-main-0\" (UID: \"45c01697-5168-4b0e-ac9b-ca821db8c611\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.487505 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.487472    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 03:15:49.622131 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.622109    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 03:15:49.623587 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:15:49.623564    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c01697_5168_4b0e_ac9b_ca821db8c611.slice/crio-be1a71371ea7738f25ac4c0205cfbdbf2facfa2558f96a033b2a91c94b0703f1 WatchSource:0}: Error finding container be1a71371ea7738f25ac4c0205cfbdbf2facfa2558f96a033b2a91c94b0703f1: Status 404 returned error can't find the container with id be1a71371ea7738f25ac4c0205cfbdbf2facfa2558f96a033b2a91c94b0703f1
Aug 04 03:15:49.844019 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:49.843986    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9353a0a-4f31-42c2-bbbf-b623049e89c3" path="/var/lib/kubelet/pods/f9353a0a-4f31-42c2-bbbf-b623049e89c3/volumes"
Aug 04 03:15:50.122733 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:50.122701    2579 generic.go:356] "Generic (PLEG): container finished" podID="45c01697-5168-4b0e-ac9b-ca821db8c611" containerID="d77305a1a8bea23aa712edb35b77d7d3c144a9437ab3cb9af1677ae1d965952f" exitCode=0
Aug 04 03:15:50.123137 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:50.122784    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45c01697-5168-4b0e-ac9b-ca821db8c611","Type":"ContainerDied","Data":"d77305a1a8bea23aa712edb35b77d7d3c144a9437ab3cb9af1677ae1d965952f"}
Aug 04 03:15:50.123137 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:50.122848    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45c01697-5168-4b0e-ac9b-ca821db8c611","Type":"ContainerStarted","Data":"be1a71371ea7738f25ac4c0205cfbdbf2facfa2558f96a033b2a91c94b0703f1"}
Aug 04 03:15:51.130085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:51.130045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45c01697-5168-4b0e-ac9b-ca821db8c611","Type":"ContainerStarted","Data":"e868b520c5e2d391acbd19e42b31f8276007ded5b8cca275ed00897d177f82fe"}
Aug 04 03:15:51.130486 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:51.130159    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45c01697-5168-4b0e-ac9b-ca821db8c611","Type":"ContainerStarted","Data":"4984d96a83cf2240d4090836b9f9fcf0cbfc8ab3f5b9d3d0f1e721a92ecd49a8"}
Aug 04 03:15:51.130486 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:51.130203    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45c01697-5168-4b0e-ac9b-ca821db8c611","Type":"ContainerStarted","Data":"dc86a0ada7b6d14f76f3546618a656f1879a1d7f194ddc12c323679f037c8fc6"}
Aug 04 03:15:51.130486 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:51.130239    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45c01697-5168-4b0e-ac9b-ca821db8c611","Type":"ContainerStarted","Data":"7556e7051555e65d16cc6373707fda1d6a7a07bbcaad504e997b4830f0f4bfee"}
Aug 04 03:15:51.130486 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:51.130276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45c01697-5168-4b0e-ac9b-ca821db8c611","Type":"ContainerStarted","Data":"44c991dd56bc9b1b5be020edb007fe4ef4d4edcb4ef83bcb335d7cd597d15b48"}
Aug 04 03:15:51.130486 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:51.130311    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45c01697-5168-4b0e-ac9b-ca821db8c611","Type":"ContainerStarted","Data":"dddbef6a0b12bcb64b4bec44d1e667ca685f1022d3316aeea0801882bd3c35ff"}
Aug 04 03:15:51.158088 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:51.158041    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.142889196 podStartE2EDuration="2.158020093s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.130897ms" isStatefulPod=false podCreationTimestamp="2026-08-04 03:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:15:51.155610347 +0000 UTC m=+381.835397466" watchObservedRunningTime="2026-08-04 03:15:51.158020093 +0000 UTC m=+381.837807211"
Aug 04 03:15:51.906185 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:51.906157    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 03:15:51.906477 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:51.906442    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="prometheus" containerID="cri-o://156afea543258758d047ba014a700221788f1e6416ab83f706fcad1c113abe5e" gracePeriod=600
Aug 04 03:15:51.906604 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:51.906470    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="thanos-sidecar" containerID="cri-o://4de3833424d99a0dc7b57a474d484d4e9d4a89cc79e2bdc87b34ce0d008d9ddc" gracePeriod=600
Aug 04 03:15:51.906604 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:51.906512    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="kube-rbac-proxy" containerID="cri-o://41b46cbbf0d197cd5cd76d78740541337ee812440796d9e8756050c5fbc25c10" gracePeriod=600
Aug 04 03:15:51.906604 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:51.906472    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="kube-rbac-proxy-web" containerID="cri-o://558f003f867682700722e68851f0d7d8d668b77779a8474b1b1a9b1793cf92c5" gracePeriod=600
Aug 04 03:15:51.906604 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:51.906477    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="config-reloader" containerID="cri-o://32e38d6f72fb89b6f49e4c491f5b71795b955286a9ec2b580981fc55378f09c9" gracePeriod=600
Aug 04 03:15:51.906813 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:51.906495    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://05a2ba741498945fa241e05a32521f8ccf72431f63a45966e15674054df5f4b9" gracePeriod=600
Aug 04 03:15:52.138793 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.138765    2579 generic.go:356] "Generic (PLEG): container finished" podID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerID="05a2ba741498945fa241e05a32521f8ccf72431f63a45966e15674054df5f4b9" exitCode=0
Aug 04 03:15:52.138793 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.138792    2579 generic.go:356] "Generic (PLEG): container finished" podID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerID="41b46cbbf0d197cd5cd76d78740541337ee812440796d9e8756050c5fbc25c10" exitCode=0
Aug 04 03:15:52.138793 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.138799    2579 generic.go:356] "Generic (PLEG): container finished" podID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerID="558f003f867682700722e68851f0d7d8d668b77779a8474b1b1a9b1793cf92c5" exitCode=0
Aug 04 03:15:52.138793 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.138804    2579 generic.go:356] "Generic (PLEG): container finished" podID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerID="4de3833424d99a0dc7b57a474d484d4e9d4a89cc79e2bdc87b34ce0d008d9ddc" exitCode=0
Aug 04 03:15:52.139263 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.138811    2579 generic.go:356] "Generic (PLEG): container finished" podID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerID="32e38d6f72fb89b6f49e4c491f5b71795b955286a9ec2b580981fc55378f09c9" exitCode=0
Aug 04 03:15:52.139263 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.138817    2579 generic.go:356] "Generic (PLEG): container finished" podID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerID="156afea543258758d047ba014a700221788f1e6416ab83f706fcad1c113abe5e" exitCode=0
Aug 04 03:15:52.139263 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.138834    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d89cee82-83f7-4b41-b3c5-344098aae10b","Type":"ContainerDied","Data":"05a2ba741498945fa241e05a32521f8ccf72431f63a45966e15674054df5f4b9"}
Aug 04 03:15:52.139263 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.138918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d89cee82-83f7-4b41-b3c5-344098aae10b","Type":"ContainerDied","Data":"41b46cbbf0d197cd5cd76d78740541337ee812440796d9e8756050c5fbc25c10"}
Aug 04 03:15:52.139263 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.139061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d89cee82-83f7-4b41-b3c5-344098aae10b","Type":"ContainerDied","Data":"558f003f867682700722e68851f0d7d8d668b77779a8474b1b1a9b1793cf92c5"}
Aug 04 03:15:52.139263 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.139092    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d89cee82-83f7-4b41-b3c5-344098aae10b","Type":"ContainerDied","Data":"4de3833424d99a0dc7b57a474d484d4e9d4a89cc79e2bdc87b34ce0d008d9ddc"}
Aug 04 03:15:52.139263 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.139122    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d89cee82-83f7-4b41-b3c5-344098aae10b","Type":"ContainerDied","Data":"32e38d6f72fb89b6f49e4c491f5b71795b955286a9ec2b580981fc55378f09c9"}
Aug 04 03:15:52.139263 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.139147    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d89cee82-83f7-4b41-b3c5-344098aae10b","Type":"ContainerDied","Data":"156afea543258758d047ba014a700221788f1e6416ab83f706fcad1c113abe5e"}
Aug 04 03:15:52.143665 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.143647    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:52.171464 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.171401    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-metrics-client-certs\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.171464 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.171433    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d89cee82-83f7-4b41-b3c5-344098aae10b-config-out\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.171464 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.171459    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-configmap-serving-certs-ca-bundle\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.171700 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.171489    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-rulefiles-1\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.171700 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.171527    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-db\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.171700 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.171553    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-web-config\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.171700 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.171576    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-trusted-ca-bundle\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.171898 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.171833    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:15:52.171981 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.171939    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:15:52.172050 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.171976    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:15:52.172050 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.172014    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-prometheus-k8s-tls\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.172050 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.172045    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-grpc-tls\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.172177 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.172065    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-configmap-metrics-client-ca\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.172177 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.172101    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-thanos-prometheus-http-client-file\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.172177 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.172137    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-rulefiles-2\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.172328 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.172183    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-kube-rbac-proxy\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.172328 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.172207    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-rulefiles-0\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.172328 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.172236    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kfmn\" (UniqueName: \"kubernetes.io/projected/d89cee82-83f7-4b41-b3c5-344098aae10b-kube-api-access-9kfmn\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.172328 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.172262    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d89cee82-83f7-4b41-b3c5-344098aae10b-tls-assets\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.172328 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.172293    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-configmap-kubelet-serving-ca-bundle\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.172686 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.172333    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-config\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.172686 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.172640    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:15:52.172686 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.172645    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.172870 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.172696    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d89cee82-83f7-4b41-b3c5-344098aae10b\" (UID: \"d89cee82-83f7-4b41-b3c5-344098aae10b\") "
Aug 04 03:15:52.173188 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.173165    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-configmap-metrics-client-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:52.173305 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.173192    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:52.173305 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.173207    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:52.173305 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.173222    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:52.174083 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.174058    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:15:52.174328 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.174295    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89cee82-83f7-4b41-b3c5-344098aae10b-config-out" (OuterVolumeSpecName: "config-out") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 03:15:52.175646 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.175217    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:15:52.177769 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.177659    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 03:15:52.177769 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.177692    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:52.177769 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.177737    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:52.178305 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.178091    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:52.178633 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.178608    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:15:52.179587 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.179564    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:52.179772 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.179742    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:52.181517 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.181476    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-config" (OuterVolumeSpecName: "config") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:52.181517 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.181493    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89cee82-83f7-4b41-b3c5-344098aae10b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:15:52.181661 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.181510    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89cee82-83f7-4b41-b3c5-344098aae10b-kube-api-access-9kfmn" (OuterVolumeSpecName: "kube-api-access-9kfmn") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "kube-api-access-9kfmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:15:52.181661 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.181640    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:52.181874 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.181854    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:52.190236 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.190208    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-web-config" (OuterVolumeSpecName: "web-config") pod "d89cee82-83f7-4b41-b3c5-344098aae10b" (UID: "d89cee82-83f7-4b41-b3c5-344098aae10b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:15:52.274344 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.274304    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-db\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:52.274344 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.274338    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-web-config\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:52.274344 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.274348    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:52.274344 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.274356    2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-grpc-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:52.274606 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.274365    2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:52.274606 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.274373    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:52.274606 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.274382    2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-kube-rbac-proxy\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:52.274606 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.274391    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:52.274606 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.274399    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kfmn\" (UniqueName: \"kubernetes.io/projected/d89cee82-83f7-4b41-b3c5-344098aae10b-kube-api-access-9kfmn\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:52.274606 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.274407    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d89cee82-83f7-4b41-b3c5-344098aae10b-tls-assets\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:52.274606 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.274417    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89cee82-83f7-4b41-b3c5-344098aae10b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:52.274606 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.274426    2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-config\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:52.274606 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.274435    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:52.274606 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.274443    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:52.274606 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.274452    2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d89cee82-83f7-4b41-b3c5-344098aae10b-secret-metrics-client-certs\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:52.274606 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:52.274460    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d89cee82-83f7-4b41-b3c5-344098aae10b-config-out\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:15:53.146230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.146198    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d89cee82-83f7-4b41-b3c5-344098aae10b","Type":"ContainerDied","Data":"7230e7af34a6c3ad0eb527994ee52f450799e6b4637b5329f9175619ab4fbfdf"}
Aug 04 03:15:53.146632 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.146256    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.146632 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.146274    2579 scope.go:117] "RemoveContainer" containerID="05a2ba741498945fa241e05a32521f8ccf72431f63a45966e15674054df5f4b9"
Aug 04 03:15:53.154646 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.154617    2579 scope.go:117] "RemoveContainer" containerID="41b46cbbf0d197cd5cd76d78740541337ee812440796d9e8756050c5fbc25c10"
Aug 04 03:15:53.161999 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.161981    2579 scope.go:117] "RemoveContainer" containerID="558f003f867682700722e68851f0d7d8d668b77779a8474b1b1a9b1793cf92c5"
Aug 04 03:15:53.168695 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.168675    2579 scope.go:117] "RemoveContainer" containerID="4de3833424d99a0dc7b57a474d484d4e9d4a89cc79e2bdc87b34ce0d008d9ddc"
Aug 04 03:15:53.173124 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.173101    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 03:15:53.182593 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.182262    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 03:15:53.184054 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.184037    2579 scope.go:117] "RemoveContainer" containerID="32e38d6f72fb89b6f49e4c491f5b71795b955286a9ec2b580981fc55378f09c9"
Aug 04 03:15:53.191659 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.191643    2579 scope.go:117] "RemoveContainer" containerID="156afea543258758d047ba014a700221788f1e6416ab83f706fcad1c113abe5e"
Aug 04 03:15:53.199314 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.199274    2579 scope.go:117] "RemoveContainer" containerID="56df7d99cacd4e5c7cca315a87373e469e2fd722db5e18bf59941edd85235cc6"
Aug 04 03:15:53.199571 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.199550    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 03:15:53.199887 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.199871    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="config-reloader"
Aug 04 03:15:53.199974 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.199889    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="config-reloader"
Aug 04 03:15:53.199974 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.199909    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="thanos-sidecar"
Aug 04 03:15:53.199974 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.199919    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="thanos-sidecar"
Aug 04 03:15:53.199974 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.199940    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="kube-rbac-proxy-web"
Aug 04 03:15:53.199974 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.199949    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="kube-rbac-proxy-web"
Aug 04 03:15:53.199974 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.199975    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="kube-rbac-proxy-thanos"
Aug 04 03:15:53.200276 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.199984    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="kube-rbac-proxy-thanos"
Aug 04 03:15:53.200276 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.200004    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="kube-rbac-proxy"
Aug 04 03:15:53.200276 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.200012    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="kube-rbac-proxy"
Aug 04 03:15:53.200276 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.200022    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="prometheus"
Aug 04 03:15:53.200276 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.200030    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="prometheus"
Aug 04 03:15:53.200276 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.200048    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="init-config-reloader"
Aug 04 03:15:53.200276 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.200057    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="init-config-reloader"
Aug 04 03:15:53.200276 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.200136    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="config-reloader"
Aug 04 03:15:53.200276 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.200151    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="kube-rbac-proxy"
Aug 04 03:15:53.200276 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.200160    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="kube-rbac-proxy-web"
Aug 04 03:15:53.200276 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.200170    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="kube-rbac-proxy-thanos"
Aug 04 03:15:53.200276 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.200179    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="thanos-sidecar"
Aug 04 03:15:53.200276 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.200187    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" containerName="prometheus"
Aug 04 03:15:53.204290 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.204269    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.206823 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.206805    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Aug 04 03:15:53.206911 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.206847    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Aug 04 03:15:53.207124 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.207107    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Aug 04 03:15:53.207198 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.207135    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Aug 04 03:15:53.207198 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.207160    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Aug 04 03:15:53.207198 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.207175    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Aug 04 03:15:53.207441 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.207424    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6jj2s\""
Aug 04 03:15:53.207500 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.207478    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Aug 04 03:15:53.207548 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.207530    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Aug 04 03:15:53.207718 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.207696    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Aug 04 03:15:53.207811 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.207800    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Aug 04 03:15:53.207921 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.207905    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Aug 04 03:15:53.208322 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.208295    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Aug 04 03:15:53.208322 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.208313    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Aug 04 03:15:53.208624 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.208570    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f5a694l9d4thg\""
Aug 04 03:15:53.209569 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.209554    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Aug 04 03:15:53.212923 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.212903    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Aug 04 03:15:53.221060 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.221034    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 03:15:53.282505 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.282474    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.282505 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.282503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.282733 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.282523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-config\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.282733 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.282538    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mg5v\" (UniqueName: \"kubernetes.io/projected/0978753a-7c25-4ff0-8a6c-236e3727b4a8-kube-api-access-2mg5v\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.282733 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.282596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.282733 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.282627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0978753a-7c25-4ff0-8a6c-236e3727b4a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.282733 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.282647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.282733 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.282665    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.282733 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.282688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.282733 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.282701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0978753a-7c25-4ff0-8a6c-236e3727b4a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.282733 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.282736    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.283165 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.282754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.283165 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.282780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0978753a-7c25-4ff0-8a6c-236e3727b4a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.283165 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.282803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.283165 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.282829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.283165 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.282893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.283165 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.282928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.283165 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.282992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.283165 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.283022    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.283165 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.283080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.384339 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.384339 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384341    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.384589 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.384589 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-config\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.384589 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mg5v\" (UniqueName: \"kubernetes.io/projected/0978753a-7c25-4ff0-8a6c-236e3727b4a8-kube-api-access-2mg5v\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.384589 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.384589 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0978753a-7c25-4ff0-8a6c-236e3727b4a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.384589 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384482    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.384589 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.384589 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.384589 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0978753a-7c25-4ff0-8a6c-236e3727b4a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.384589 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.385086 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.385086 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0978753a-7c25-4ff0-8a6c-236e3727b4a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.385086 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.385086 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.385086 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.385086 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.385086 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384840    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.385086 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.384866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.385505 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.385190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.385505 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.385374    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.387615 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.385772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.387615 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.386664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.387615 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.387285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.387615 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.387550    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-config\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.387615 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.387613    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.387894 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.387736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.387981 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.387946    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.388396 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.388225    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0978753a-7c25-4ff0-8a6c-236e3727b4a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.388396 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.388351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0978753a-7c25-4ff0-8a6c-236e3727b4a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.388537 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.388454    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.388716 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.388692    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.389232 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.389207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.389829 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.389801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0978753a-7c25-4ff0-8a6c-236e3727b4a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.390344 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.390321    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.390588 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.390561    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.390672 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.390643    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0978753a-7c25-4ff0-8a6c-236e3727b4a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.391683 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.391658    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0978753a-7c25-4ff0-8a6c-236e3727b4a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.392185 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.392167    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mg5v\" (UniqueName: \"kubernetes.io/projected/0978753a-7c25-4ff0-8a6c-236e3727b4a8-kube-api-access-2mg5v\") pod \"prometheus-k8s-0\" (UID: \"0978753a-7c25-4ff0-8a6c-236e3727b4a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.515389 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.515353    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:15:53.649850 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.649820    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 03:15:53.652640 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:15:53.652617    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0978753a_7c25_4ff0_8a6c_236e3727b4a8.slice/crio-735f915843a518b60e3832c63319c21f4e6c0087f6c56af1b06d3d20f21c62b7 WatchSource:0}: Error finding container 735f915843a518b60e3832c63319c21f4e6c0087f6c56af1b06d3d20f21c62b7: Status 404 returned error can't find the container with id 735f915843a518b60e3832c63319c21f4e6c0087f6c56af1b06d3d20f21c62b7
Aug 04 03:15:53.844029 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:53.843935    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d89cee82-83f7-4b41-b3c5-344098aae10b" path="/var/lib/kubelet/pods/d89cee82-83f7-4b41-b3c5-344098aae10b/volumes"
Aug 04 03:15:54.153262 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:54.153179    2579 generic.go:356] "Generic (PLEG): container finished" podID="0978753a-7c25-4ff0-8a6c-236e3727b4a8" containerID="c370f127bbf3fa78b3e18a7d942c4e214ff656f937f16a38ca20bfda8c9f9b65" exitCode=0
Aug 04 03:15:54.153262 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:54.153243    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0978753a-7c25-4ff0-8a6c-236e3727b4a8","Type":"ContainerDied","Data":"c370f127bbf3fa78b3e18a7d942c4e214ff656f937f16a38ca20bfda8c9f9b65"}
Aug 04 03:15:54.153631 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:54.153299    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0978753a-7c25-4ff0-8a6c-236e3727b4a8","Type":"ContainerStarted","Data":"735f915843a518b60e3832c63319c21f4e6c0087f6c56af1b06d3d20f21c62b7"}
Aug 04 03:15:55.164902 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:55.164868    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0978753a-7c25-4ff0-8a6c-236e3727b4a8","Type":"ContainerStarted","Data":"b2c3df35c26d96332add183411855ee848d4a909eabbc01ad28e210c9b28ebdc"}
Aug 04 03:15:55.165412 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:55.164955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0978753a-7c25-4ff0-8a6c-236e3727b4a8","Type":"ContainerStarted","Data":"b44ca54e1073660bceddb7eede9738e53537962520bce281ac65d4c4a556b524"}
Aug 04 03:15:55.165412 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:55.165004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0978753a-7c25-4ff0-8a6c-236e3727b4a8","Type":"ContainerStarted","Data":"8b350621d6cb9a94c755ef3d8d5f26b89c45aede2a3c2251cd8cf95f01df53c8"}
Aug 04 03:15:55.165412 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:55.165042    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0978753a-7c25-4ff0-8a6c-236e3727b4a8","Type":"ContainerStarted","Data":"1f98b624abd8818882429a78c7b97016d6f5b8d21d5bc345ff5f1655e28d5a18"}
Aug 04 03:15:55.165412 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:55.165071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0978753a-7c25-4ff0-8a6c-236e3727b4a8","Type":"ContainerStarted","Data":"16dad9e03cf1c8037ef91c60a293ec540602e34daaef6e06e9ff0fb96c37cae3"}
Aug 04 03:15:55.165412 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:55.165096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0978753a-7c25-4ff0-8a6c-236e3727b4a8","Type":"ContainerStarted","Data":"d69ab9602e79a0f8eab65761c39113e6b23559f8830ec4c5234c6f88473353bd"}
Aug 04 03:15:55.199434 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:55.199375    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.180784557 podStartE2EDuration="2.19935877s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.574213ms" isStatefulPod=false podCreationTimestamp="2026-08-04 03:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:15:55.197066152 +0000 UTC m=+385.876853298" watchObservedRunningTime="2026-08-04 03:15:55.19935877 +0000 UTC m=+385.879145888"
Aug 04 03:15:58.516484 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:15:58.516447    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:16:08.349795 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.349759    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d775bd59-hd6gq"]
Aug 04 03:16:08.354672 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.354651    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.364489 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.364463    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d775bd59-hd6gq"]
Aug 04 03:16:08.410241 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.410214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-oauth-serving-cert\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.410371 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.410243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-console-config\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.410371 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.410270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32cadeb2-cb8a-46e5-9f33-fde888a05973-console-oauth-config\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.410371 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.410340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-trusted-ca-bundle\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.410475 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.410390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32cadeb2-cb8a-46e5-9f33-fde888a05973-console-serving-cert\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.410475 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.410404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-service-ca\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.410475 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.410425    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb72s\" (UniqueName: \"kubernetes.io/projected/32cadeb2-cb8a-46e5-9f33-fde888a05973-kube-api-access-vb72s\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.511586 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.511539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32cadeb2-cb8a-46e5-9f33-fde888a05973-console-oauth-config\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.511586 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.511597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-trusted-ca-bundle\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.511816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.511640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32cadeb2-cb8a-46e5-9f33-fde888a05973-console-serving-cert\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.511816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.511654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-service-ca\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.511816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.511678    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb72s\" (UniqueName: \"kubernetes.io/projected/32cadeb2-cb8a-46e5-9f33-fde888a05973-kube-api-access-vb72s\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.511816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.511720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-oauth-serving-cert\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.511816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.511746    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-console-config\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.512504 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.512478    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-console-config\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.512504 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.512496    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-oauth-serving-cert\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.512689 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.512666    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-service-ca\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.512996 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.512976    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-trusted-ca-bundle\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.514205 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.514174    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32cadeb2-cb8a-46e5-9f33-fde888a05973-console-serving-cert\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.514309 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.514223    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32cadeb2-cb8a-46e5-9f33-fde888a05973-console-oauth-config\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.521912 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.521891    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb72s\" (UniqueName: \"kubernetes.io/projected/32cadeb2-cb8a-46e5-9f33-fde888a05973-kube-api-access-vb72s\") pod \"console-5d775bd59-hd6gq\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") " pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.666024 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.665933    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:08.797131 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:08.797063    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d775bd59-hd6gq"]
Aug 04 03:16:08.799177 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:16:08.799149    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32cadeb2_cb8a_46e5_9f33_fde888a05973.slice/crio-e09e9a51104596b212d8bf8949ba878d089d665e5fd72454dd917c9d7b9d48fb WatchSource:0}: Error finding container e09e9a51104596b212d8bf8949ba878d089d665e5fd72454dd917c9d7b9d48fb: Status 404 returned error can't find the container with id e09e9a51104596b212d8bf8949ba878d089d665e5fd72454dd917c9d7b9d48fb
Aug 04 03:16:09.221577 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:09.221542    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d775bd59-hd6gq" event={"ID":"32cadeb2-cb8a-46e5-9f33-fde888a05973","Type":"ContainerStarted","Data":"5ed7034d20958858dbeb3d9617ceba377dd2c19d4520375ec268df42c5766950"}
Aug 04 03:16:09.221761 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:09.221591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d775bd59-hd6gq" event={"ID":"32cadeb2-cb8a-46e5-9f33-fde888a05973","Type":"ContainerStarted","Data":"e09e9a51104596b212d8bf8949ba878d089d665e5fd72454dd917c9d7b9d48fb"}
Aug 04 03:16:09.241512 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:09.241462    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d775bd59-hd6gq" podStartSLOduration=1.241446428 podStartE2EDuration="1.241446428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:16:09.239285954 +0000 UTC m=+399.919073100" watchObservedRunningTime="2026-08-04 03:16:09.241446428 +0000 UTC m=+399.921233548"
Aug 04 03:16:18.666854 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:18.666817    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:18.667369 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:18.666873    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:18.671455 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:18.671432    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:19.264696 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:19.264670    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:16:19.306595 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:19.306568    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-569999c877-gdfxj"]
Aug 04 03:16:44.327388 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.327327    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-569999c877-gdfxj" podUID="a4afb1aa-1736-4e99-bedb-e28c5de8ab5e" containerName="console" containerID="cri-o://93b0dd70bd6af7743098bf20379644e6a5bb923279127c8e32541e4a416e7fc8" gracePeriod=15
Aug 04 03:16:44.570802 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.570778    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:16:44.720529 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.720497    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhrgt\" (UniqueName: \"kubernetes.io/projected/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-kube-api-access-nhrgt\") pod \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") "
Aug 04 03:16:44.720723 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.720549    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-service-ca\") pod \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") "
Aug 04 03:16:44.720723 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.720579    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-console-oauth-config\") pod \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") "
Aug 04 03:16:44.720723 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.720610    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-oauth-serving-cert\") pod \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") "
Aug 04 03:16:44.720723 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.720649    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-console-config\") pod \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") "
Aug 04 03:16:44.720723 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.720683    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-console-serving-cert\") pod \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") "
Aug 04 03:16:44.720723 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.720707    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-trusted-ca-bundle\") pod \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\" (UID: \"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e\") "
Aug 04 03:16:44.721071 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.721024    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-service-ca" (OuterVolumeSpecName: "service-ca") pod "a4afb1aa-1736-4e99-bedb-e28c5de8ab5e" (UID: "a4afb1aa-1736-4e99-bedb-e28c5de8ab5e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:16:44.721167 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.721142    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a4afb1aa-1736-4e99-bedb-e28c5de8ab5e" (UID: "a4afb1aa-1736-4e99-bedb-e28c5de8ab5e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:16:44.721211 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.721168    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-console-config" (OuterVolumeSpecName: "console-config") pod "a4afb1aa-1736-4e99-bedb-e28c5de8ab5e" (UID: "a4afb1aa-1736-4e99-bedb-e28c5de8ab5e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:16:44.721252 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.721230    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a4afb1aa-1736-4e99-bedb-e28c5de8ab5e" (UID: "a4afb1aa-1736-4e99-bedb-e28c5de8ab5e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:16:44.722710 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.722681    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-kube-api-access-nhrgt" (OuterVolumeSpecName: "kube-api-access-nhrgt") pod "a4afb1aa-1736-4e99-bedb-e28c5de8ab5e" (UID: "a4afb1aa-1736-4e99-bedb-e28c5de8ab5e"). InnerVolumeSpecName "kube-api-access-nhrgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:16:44.723171 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.723149    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a4afb1aa-1736-4e99-bedb-e28c5de8ab5e" (UID: "a4afb1aa-1736-4e99-bedb-e28c5de8ab5e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:16:44.723242 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.723176    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a4afb1aa-1736-4e99-bedb-e28c5de8ab5e" (UID: "a4afb1aa-1736-4e99-bedb-e28c5de8ab5e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:16:44.822120 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.822084    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-oauth-serving-cert\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:16:44.822120 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.822115    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-console-config\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:16:44.822120 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.822128    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-console-serving-cert\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:16:44.822361 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.822140    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-trusted-ca-bundle\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:16:44.822361 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.822151    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhrgt\" (UniqueName: \"kubernetes.io/projected/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-kube-api-access-nhrgt\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:16:44.822361 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.822163    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:16:44.822361 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:44.822173    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e-console-oauth-config\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:16:45.366842 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:45.366808    2579 generic.go:356] "Generic (PLEG): container finished" podID="a4afb1aa-1736-4e99-bedb-e28c5de8ab5e" containerID="93b0dd70bd6af7743098bf20379644e6a5bb923279127c8e32541e4a416e7fc8" exitCode=0
Aug 04 03:16:45.367296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:45.366854    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-569999c877-gdfxj"
Aug 04 03:16:45.367296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:45.366863    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-569999c877-gdfxj" event={"ID":"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e","Type":"ContainerDied","Data":"93b0dd70bd6af7743098bf20379644e6a5bb923279127c8e32541e4a416e7fc8"}
Aug 04 03:16:45.367296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:45.366903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-569999c877-gdfxj" event={"ID":"a4afb1aa-1736-4e99-bedb-e28c5de8ab5e","Type":"ContainerDied","Data":"c6a9393a7cbe2c259809d283635c9936eb41c67dffbedd817b81cb04b9aec442"}
Aug 04 03:16:45.367296 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:45.366923    2579 scope.go:117] "RemoveContainer" containerID="93b0dd70bd6af7743098bf20379644e6a5bb923279127c8e32541e4a416e7fc8"
Aug 04 03:16:45.375245 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:45.375227    2579 scope.go:117] "RemoveContainer" containerID="93b0dd70bd6af7743098bf20379644e6a5bb923279127c8e32541e4a416e7fc8"
Aug 04 03:16:45.375517 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:16:45.375498    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93b0dd70bd6af7743098bf20379644e6a5bb923279127c8e32541e4a416e7fc8\": container with ID starting with 93b0dd70bd6af7743098bf20379644e6a5bb923279127c8e32541e4a416e7fc8 not found: ID does not exist" containerID="93b0dd70bd6af7743098bf20379644e6a5bb923279127c8e32541e4a416e7fc8"
Aug 04 03:16:45.375580 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:45.375529    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b0dd70bd6af7743098bf20379644e6a5bb923279127c8e32541e4a416e7fc8"} err="failed to get container status \"93b0dd70bd6af7743098bf20379644e6a5bb923279127c8e32541e4a416e7fc8\": rpc error: code = NotFound desc = could not find container \"93b0dd70bd6af7743098bf20379644e6a5bb923279127c8e32541e4a416e7fc8\": container with ID starting with 93b0dd70bd6af7743098bf20379644e6a5bb923279127c8e32541e4a416e7fc8 not found: ID does not exist"
Aug 04 03:16:45.388032 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:45.388009    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-569999c877-gdfxj"]
Aug 04 03:16:45.393807 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:45.393788    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-569999c877-gdfxj"]
Aug 04 03:16:45.844460 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:45.844433    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4afb1aa-1736-4e99-bedb-e28c5de8ab5e" path="/var/lib/kubelet/pods/a4afb1aa-1736-4e99-bedb-e28c5de8ab5e/volumes"
Aug 04 03:16:53.515791 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:53.515707    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:16:53.531091 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:53.531067    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:16:54.422224 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:16:54.422198    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 03:17:42.445807 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:42.445771    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ttsr4"]
Aug 04 03:17:42.446287 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:42.446105    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4afb1aa-1736-4e99-bedb-e28c5de8ab5e" containerName="console"
Aug 04 03:17:42.446287 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:42.446116    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4afb1aa-1736-4e99-bedb-e28c5de8ab5e" containerName="console"
Aug 04 03:17:42.446287 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:42.446170    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4afb1aa-1736-4e99-bedb-e28c5de8ab5e" containerName="console"
Aug 04 03:17:42.450188 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:42.450171    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ttsr4"
Aug 04 03:17:42.452430 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:42.452406    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Aug 04 03:17:42.452430 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:42.452421    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-fzlkg\""
Aug 04 03:17:42.453351 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:42.453337    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Aug 04 03:17:42.459275 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:42.459252    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ttsr4"]
Aug 04 03:17:42.514666 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:42.514634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e812fe22-ab0b-46a9-956b-a863f5d340f2-bound-sa-token\") pod \"cert-manager-55b4d954d5-ttsr4\" (UID: \"e812fe22-ab0b-46a9-956b-a863f5d340f2\") " pod="cert-manager/cert-manager-55b4d954d5-ttsr4"
Aug 04 03:17:42.514818 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:42.514675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kdcd\" (UniqueName: \"kubernetes.io/projected/e812fe22-ab0b-46a9-956b-a863f5d340f2-kube-api-access-9kdcd\") pod \"cert-manager-55b4d954d5-ttsr4\" (UID: \"e812fe22-ab0b-46a9-956b-a863f5d340f2\") " pod="cert-manager/cert-manager-55b4d954d5-ttsr4"
Aug 04 03:17:42.615771 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:42.615738    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e812fe22-ab0b-46a9-956b-a863f5d340f2-bound-sa-token\") pod \"cert-manager-55b4d954d5-ttsr4\" (UID: \"e812fe22-ab0b-46a9-956b-a863f5d340f2\") " pod="cert-manager/cert-manager-55b4d954d5-ttsr4"
Aug 04 03:17:42.615935 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:42.615780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kdcd\" (UniqueName: \"kubernetes.io/projected/e812fe22-ab0b-46a9-956b-a863f5d340f2-kube-api-access-9kdcd\") pod \"cert-manager-55b4d954d5-ttsr4\" (UID: \"e812fe22-ab0b-46a9-956b-a863f5d340f2\") " pod="cert-manager/cert-manager-55b4d954d5-ttsr4"
Aug 04 03:17:42.623768 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:42.623747    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kdcd\" (UniqueName: \"kubernetes.io/projected/e812fe22-ab0b-46a9-956b-a863f5d340f2-kube-api-access-9kdcd\") pod \"cert-manager-55b4d954d5-ttsr4\" (UID: \"e812fe22-ab0b-46a9-956b-a863f5d340f2\") " pod="cert-manager/cert-manager-55b4d954d5-ttsr4"
Aug 04 03:17:42.623907 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:42.623886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e812fe22-ab0b-46a9-956b-a863f5d340f2-bound-sa-token\") pod \"cert-manager-55b4d954d5-ttsr4\" (UID: \"e812fe22-ab0b-46a9-956b-a863f5d340f2\") " pod="cert-manager/cert-manager-55b4d954d5-ttsr4"
Aug 04 03:17:42.777912 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:42.777879    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ttsr4"
Aug 04 03:17:42.916540 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:42.916495    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ttsr4"]
Aug 04 03:17:42.920314 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:17:42.920291    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode812fe22_ab0b_46a9_956b_a863f5d340f2.slice/crio-e28049b41a16070af57d7354e92b7cdcd1682d4447d332e44705403b6b524543 WatchSource:0}: Error finding container e28049b41a16070af57d7354e92b7cdcd1682d4447d332e44705403b6b524543: Status 404 returned error can't find the container with id e28049b41a16070af57d7354e92b7cdcd1682d4447d332e44705403b6b524543
Aug 04 03:17:43.610444 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:43.610407    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ttsr4" event={"ID":"e812fe22-ab0b-46a9-956b-a863f5d340f2","Type":"ContainerStarted","Data":"e28049b41a16070af57d7354e92b7cdcd1682d4447d332e44705403b6b524543"}
Aug 04 03:17:46.626443 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:46.626358    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ttsr4" event={"ID":"e812fe22-ab0b-46a9-956b-a863f5d340f2","Type":"ContainerStarted","Data":"08c5c03f39390deb6dc08508793b835d0586943f8345b70d36b718fc318fd963"}
Aug 04 03:17:46.642509 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:17:46.642461    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-ttsr4" podStartSLOduration=1.28153307 podStartE2EDuration="4.64244645s" totalImagesPullingTime="3.36091338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:17:46.64118283 +0000 UTC m=+497.320969961" watchObservedRunningTime="2026-08-04 03:17:46.64244645 +0000 UTC m=+497.322233569"
Aug 04 03:18:04.130056 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.130022    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"]
Aug 04 03:18:04.133538 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.133515    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"
Aug 04 03:18:04.139004 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.138981    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Aug 04 03:18:04.139226 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.138996    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Aug 04 03:18:04.139312 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.139080    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Aug 04 03:18:04.139312 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.139120    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Aug 04 03:18:04.140174 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.139996    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cbjbp\""
Aug 04 03:18:04.148180 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.148154    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"]
Aug 04 03:18:04.196843 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.196806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c239d70-6373-43eb-acd1-2fb0a5631d82-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx\" (UID: \"3c239d70-6373-43eb-acd1-2fb0a5631d82\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"
Aug 04 03:18:04.196843 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.196846    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c239d70-6373-43eb-acd1-2fb0a5631d82-tmp\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx\" (UID: \"3c239d70-6373-43eb-acd1-2fb0a5631d82\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"
Aug 04 03:18:04.197085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.196867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c239d70-6373-43eb-acd1-2fb0a5631d82-manifests\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx\" (UID: \"3c239d70-6373-43eb-acd1-2fb0a5631d82\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"
Aug 04 03:18:04.197085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.196924    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkx5q\" (UniqueName: \"kubernetes.io/projected/3c239d70-6373-43eb-acd1-2fb0a5631d82-kube-api-access-tkx5q\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx\" (UID: \"3c239d70-6373-43eb-acd1-2fb0a5631d82\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"
Aug 04 03:18:04.197085 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.197003    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c239d70-6373-43eb-acd1-2fb0a5631d82-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx\" (UID: \"3c239d70-6373-43eb-acd1-2fb0a5631d82\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"
Aug 04 03:18:04.298161 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.298119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c239d70-6373-43eb-acd1-2fb0a5631d82-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx\" (UID: \"3c239d70-6373-43eb-acd1-2fb0a5631d82\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"
Aug 04 03:18:04.298161 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.298163    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c239d70-6373-43eb-acd1-2fb0a5631d82-tmp\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx\" (UID: \"3c239d70-6373-43eb-acd1-2fb0a5631d82\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"
Aug 04 03:18:04.298402 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.298193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c239d70-6373-43eb-acd1-2fb0a5631d82-manifests\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx\" (UID: \"3c239d70-6373-43eb-acd1-2fb0a5631d82\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"
Aug 04 03:18:04.298402 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.298223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkx5q\" (UniqueName: \"kubernetes.io/projected/3c239d70-6373-43eb-acd1-2fb0a5631d82-kube-api-access-tkx5q\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx\" (UID: \"3c239d70-6373-43eb-acd1-2fb0a5631d82\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"
Aug 04 03:18:04.298402 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.298265    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c239d70-6373-43eb-acd1-2fb0a5631d82-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx\" (UID: \"3c239d70-6373-43eb-acd1-2fb0a5631d82\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"
Aug 04 03:18:04.298684 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.298660    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c239d70-6373-43eb-acd1-2fb0a5631d82-manifests\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx\" (UID: \"3c239d70-6373-43eb-acd1-2fb0a5631d82\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"
Aug 04 03:18:04.300518 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.300491    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c239d70-6373-43eb-acd1-2fb0a5631d82-tmp\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx\" (UID: \"3c239d70-6373-43eb-acd1-2fb0a5631d82\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"
Aug 04 03:18:04.300606 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.300581    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c239d70-6373-43eb-acd1-2fb0a5631d82-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx\" (UID: \"3c239d70-6373-43eb-acd1-2fb0a5631d82\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"
Aug 04 03:18:04.300729 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.300713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c239d70-6373-43eb-acd1-2fb0a5631d82-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx\" (UID: \"3c239d70-6373-43eb-acd1-2fb0a5631d82\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"
Aug 04 03:18:04.305564 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.305543    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkx5q\" (UniqueName: \"kubernetes.io/projected/3c239d70-6373-43eb-acd1-2fb0a5631d82-kube-api-access-tkx5q\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx\" (UID: \"3c239d70-6373-43eb-acd1-2fb0a5631d82\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"
Aug 04 03:18:04.443236 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.443151    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"
Aug 04 03:18:04.578984 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.578940    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"]
Aug 04 03:18:04.582255 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:18:04.582224    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c239d70_6373_43eb_acd1_2fb0a5631d82.slice/crio-1408e13daa650637a502312beb04bf265f16ef3bcead95ca3e520ebefb6404c4 WatchSource:0}: Error finding container 1408e13daa650637a502312beb04bf265f16ef3bcead95ca3e520ebefb6404c4: Status 404 returned error can't find the container with id 1408e13daa650637a502312beb04bf265f16ef3bcead95ca3e520ebefb6404c4
Aug 04 03:18:04.703406 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:04.703315    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx" event={"ID":"3c239d70-6373-43eb-acd1-2fb0a5631d82","Type":"ContainerStarted","Data":"1408e13daa650637a502312beb04bf265f16ef3bcead95ca3e520ebefb6404c4"}
Aug 04 03:18:07.721341 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:07.721301    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx" event={"ID":"3c239d70-6373-43eb-acd1-2fb0a5631d82","Type":"ContainerStarted","Data":"fcebb3d2e2925ec1af073d5561b6be98e4cfbdcb18db030517e27ea4ae5b1955"}
Aug 04 03:18:09.738538 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:09.738496    2579 generic.go:356] "Generic (PLEG): container finished" podID="3c239d70-6373-43eb-acd1-2fb0a5631d82" containerID="fcebb3d2e2925ec1af073d5561b6be98e4cfbdcb18db030517e27ea4ae5b1955" exitCode=0
Aug 04 03:18:09.739057 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:09.738617    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx" event={"ID":"3c239d70-6373-43eb-acd1-2fb0a5631d82","Type":"ContainerDied","Data":"fcebb3d2e2925ec1af073d5561b6be98e4cfbdcb18db030517e27ea4ae5b1955"}
Aug 04 03:18:10.744265 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:10.744231    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx" event={"ID":"3c239d70-6373-43eb-acd1-2fb0a5631d82","Type":"ContainerStarted","Data":"565e4cdf4a326031fed6062ff12c5e8fe103b1cdf898e297d91138e48bc945db"}
Aug 04 03:18:10.744703 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:10.744453    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"
Aug 04 03:18:10.766381 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:10.766333    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx" podStartSLOduration=2.125119777 podStartE2EDuration="6.76631807s" totalImagesPullingTime="3.608765992s" totalInitContainerRuntime="1.032432301s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:18:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:18:10.765510148 +0000 UTC m=+521.445297288" watchObservedRunningTime="2026-08-04 03:18:10.76631807 +0000 UTC m=+521.446105188"
Aug 04 03:18:21.751185 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:21.751100    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx"
Aug 04 03:18:23.971341 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:23.971309    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr"]
Aug 04 03:18:23.974548 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:23.974529    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr"
Aug 04 03:18:23.977088 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:23.976943    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Aug 04 03:18:23.977226 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:23.977102    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Aug 04 03:18:23.978000 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:23.977975    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Aug 04 03:18:23.978000 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:23.977992    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Aug 04 03:18:23.978000 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:23.977998    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Aug 04 03:18:23.978196 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:23.978041    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-622s4\""
Aug 04 03:18:23.985471 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:23.985450    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr"]
Aug 04 03:18:24.077253 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:24.077214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/90dafb77-eb17-46d6-9ffa-1e6568c6a439-manager-config\") pod \"lws-controller-manager-6488bdc59d-gmpnr\" (UID: \"90dafb77-eb17-46d6-9ffa-1e6568c6a439\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr"
Aug 04 03:18:24.077499 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:24.077258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/90dafb77-eb17-46d6-9ffa-1e6568c6a439-metrics-cert\") pod \"lws-controller-manager-6488bdc59d-gmpnr\" (UID: \"90dafb77-eb17-46d6-9ffa-1e6568c6a439\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr"
Aug 04 03:18:24.077499 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:24.077326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgmjg\" (UniqueName: \"kubernetes.io/projected/90dafb77-eb17-46d6-9ffa-1e6568c6a439-kube-api-access-zgmjg\") pod \"lws-controller-manager-6488bdc59d-gmpnr\" (UID: \"90dafb77-eb17-46d6-9ffa-1e6568c6a439\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr"
Aug 04 03:18:24.077499 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:24.077432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90dafb77-eb17-46d6-9ffa-1e6568c6a439-cert\") pod \"lws-controller-manager-6488bdc59d-gmpnr\" (UID: \"90dafb77-eb17-46d6-9ffa-1e6568c6a439\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr"
Aug 04 03:18:24.178802 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:24.178760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90dafb77-eb17-46d6-9ffa-1e6568c6a439-cert\") pod \"lws-controller-manager-6488bdc59d-gmpnr\" (UID: \"90dafb77-eb17-46d6-9ffa-1e6568c6a439\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr"
Aug 04 03:18:24.179042 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:24.178844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/90dafb77-eb17-46d6-9ffa-1e6568c6a439-manager-config\") pod \"lws-controller-manager-6488bdc59d-gmpnr\" (UID: \"90dafb77-eb17-46d6-9ffa-1e6568c6a439\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr"
Aug 04 03:18:24.179042 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:24.178862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/90dafb77-eb17-46d6-9ffa-1e6568c6a439-metrics-cert\") pod \"lws-controller-manager-6488bdc59d-gmpnr\" (UID: \"90dafb77-eb17-46d6-9ffa-1e6568c6a439\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr"
Aug 04 03:18:24.179042 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:24.178882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgmjg\" (UniqueName: \"kubernetes.io/projected/90dafb77-eb17-46d6-9ffa-1e6568c6a439-kube-api-access-zgmjg\") pod \"lws-controller-manager-6488bdc59d-gmpnr\" (UID: \"90dafb77-eb17-46d6-9ffa-1e6568c6a439\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr"
Aug 04 03:18:24.179600 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:24.179579    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/90dafb77-eb17-46d6-9ffa-1e6568c6a439-manager-config\") pod \"lws-controller-manager-6488bdc59d-gmpnr\" (UID: \"90dafb77-eb17-46d6-9ffa-1e6568c6a439\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr"
Aug 04 03:18:24.181402 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:24.181382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/90dafb77-eb17-46d6-9ffa-1e6568c6a439-metrics-cert\") pod \"lws-controller-manager-6488bdc59d-gmpnr\" (UID: \"90dafb77-eb17-46d6-9ffa-1e6568c6a439\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr"
Aug 04 03:18:24.181502 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:24.181404    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90dafb77-eb17-46d6-9ffa-1e6568c6a439-cert\") pod \"lws-controller-manager-6488bdc59d-gmpnr\" (UID: \"90dafb77-eb17-46d6-9ffa-1e6568c6a439\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr"
Aug 04 03:18:24.192898 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:24.192870    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgmjg\" (UniqueName: \"kubernetes.io/projected/90dafb77-eb17-46d6-9ffa-1e6568c6a439-kube-api-access-zgmjg\") pod \"lws-controller-manager-6488bdc59d-gmpnr\" (UID: \"90dafb77-eb17-46d6-9ffa-1e6568c6a439\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr"
Aug 04 03:18:24.284704 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:24.284608    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr"
Aug 04 03:18:24.421616 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:24.421593    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr"]
Aug 04 03:18:24.423861 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:18:24.423835    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90dafb77_eb17_46d6_9ffa_1e6568c6a439.slice/crio-b5a9e387f876a373a76296e0cf5f0991a3a44040d32342107236b4c9d20f19f7 WatchSource:0}: Error finding container b5a9e387f876a373a76296e0cf5f0991a3a44040d32342107236b4c9d20f19f7: Status 404 returned error can't find the container with id b5a9e387f876a373a76296e0cf5f0991a3a44040d32342107236b4c9d20f19f7
Aug 04 03:18:24.803816 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:24.803784    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr" event={"ID":"90dafb77-eb17-46d6-9ffa-1e6568c6a439","Type":"ContainerStarted","Data":"b5a9e387f876a373a76296e0cf5f0991a3a44040d32342107236b4c9d20f19f7"}
Aug 04 03:18:26.880318 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:26.880284    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-69dbcc4dfb-lmwkc"]
Aug 04 03:18:26.884063 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:26.884035    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69dbcc4dfb-lmwkc"
Aug 04 03:18:26.886521 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:26.886477    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Aug 04 03:18:26.886671 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:26.886561    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Aug 04 03:18:26.887634 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:26.887599    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Aug 04 03:18:26.887634 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:26.887617    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Aug 04 03:18:26.887804 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:26.887657    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-h72rg\""
Aug 04 03:18:26.892811 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:26.892790    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69dbcc4dfb-lmwkc"]
Aug 04 03:18:27.008423 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:27.008368    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4732185c-0a57-4163-b84b-89c007fbed25-tmp\") pod \"kube-auth-proxy-69dbcc4dfb-lmwkc\" (UID: \"4732185c-0a57-4163-b84b-89c007fbed25\") " pod="openshift-ingress/kube-auth-proxy-69dbcc4dfb-lmwkc"
Aug 04 03:18:27.008618 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:27.008432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4xxx\" (UniqueName: \"kubernetes.io/projected/4732185c-0a57-4163-b84b-89c007fbed25-kube-api-access-f4xxx\") pod \"kube-auth-proxy-69dbcc4dfb-lmwkc\" (UID: \"4732185c-0a57-4163-b84b-89c007fbed25\") " pod="openshift-ingress/kube-auth-proxy-69dbcc4dfb-lmwkc"
Aug 04 03:18:27.008618 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:27.008586    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4732185c-0a57-4163-b84b-89c007fbed25-tls-certs\") pod \"kube-auth-proxy-69dbcc4dfb-lmwkc\" (UID: \"4732185c-0a57-4163-b84b-89c007fbed25\") " pod="openshift-ingress/kube-auth-proxy-69dbcc4dfb-lmwkc"
Aug 04 03:18:27.109252 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:27.109230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4732185c-0a57-4163-b84b-89c007fbed25-tmp\") pod \"kube-auth-proxy-69dbcc4dfb-lmwkc\" (UID: \"4732185c-0a57-4163-b84b-89c007fbed25\") " pod="openshift-ingress/kube-auth-proxy-69dbcc4dfb-lmwkc"
Aug 04 03:18:27.109402 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:27.109257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4xxx\" (UniqueName: \"kubernetes.io/projected/4732185c-0a57-4163-b84b-89c007fbed25-kube-api-access-f4xxx\") pod \"kube-auth-proxy-69dbcc4dfb-lmwkc\" (UID: \"4732185c-0a57-4163-b84b-89c007fbed25\") " pod="openshift-ingress/kube-auth-proxy-69dbcc4dfb-lmwkc"
Aug 04 03:18:27.109402 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:27.109295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4732185c-0a57-4163-b84b-89c007fbed25-tls-certs\") pod \"kube-auth-proxy-69dbcc4dfb-lmwkc\" (UID: \"4732185c-0a57-4163-b84b-89c007fbed25\") " pod="openshift-ingress/kube-auth-proxy-69dbcc4dfb-lmwkc"
Aug 04 03:18:27.111548 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:27.111525    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4732185c-0a57-4163-b84b-89c007fbed25-tmp\") pod \"kube-auth-proxy-69dbcc4dfb-lmwkc\" (UID: \"4732185c-0a57-4163-b84b-89c007fbed25\") " pod="openshift-ingress/kube-auth-proxy-69dbcc4dfb-lmwkc"
Aug 04 03:18:27.111728 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:27.111710    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4732185c-0a57-4163-b84b-89c007fbed25-tls-certs\") pod \"kube-auth-proxy-69dbcc4dfb-lmwkc\" (UID: \"4732185c-0a57-4163-b84b-89c007fbed25\") " pod="openshift-ingress/kube-auth-proxy-69dbcc4dfb-lmwkc"
Aug 04 03:18:27.117343 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:27.117325    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4xxx\" (UniqueName: \"kubernetes.io/projected/4732185c-0a57-4163-b84b-89c007fbed25-kube-api-access-f4xxx\") pod \"kube-auth-proxy-69dbcc4dfb-lmwkc\" (UID: \"4732185c-0a57-4163-b84b-89c007fbed25\") " pod="openshift-ingress/kube-auth-proxy-69dbcc4dfb-lmwkc"
Aug 04 03:18:27.198724 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:27.198452    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69dbcc4dfb-lmwkc"
Aug 04 03:18:27.338406 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:27.338339    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69dbcc4dfb-lmwkc"]
Aug 04 03:18:27.341138 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:18:27.341109    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4732185c_0a57_4163_b84b_89c007fbed25.slice/crio-e3fc32478c42d906e08469ec35039e7d08906d4efc903667e1b8c43330c4011e WatchSource:0}: Error finding container e3fc32478c42d906e08469ec35039e7d08906d4efc903667e1b8c43330c4011e: Status 404 returned error can't find the container with id e3fc32478c42d906e08469ec35039e7d08906d4efc903667e1b8c43330c4011e
Aug 04 03:18:27.824918 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:27.824871    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69dbcc4dfb-lmwkc" event={"ID":"4732185c-0a57-4163-b84b-89c007fbed25","Type":"ContainerStarted","Data":"e3fc32478c42d906e08469ec35039e7d08906d4efc903667e1b8c43330c4011e"}
Aug 04 03:18:27.826419 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:27.826394    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr" event={"ID":"90dafb77-eb17-46d6-9ffa-1e6568c6a439","Type":"ContainerStarted","Data":"6e663766611e364e93c14234dd40efcb561d384462da85da24dcfc4a7d61c268"}
Aug 04 03:18:27.826555 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:27.826514    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr"
Aug 04 03:18:27.849767 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:27.849717    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr" podStartSLOduration=2.166415281 podStartE2EDuration="4.849701193s" totalImagesPullingTime="2.683285912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:18:27.848845479 +0000 UTC m=+538.528632714" watchObservedRunningTime="2026-08-04 03:18:27.849701193 +0000 UTC m=+538.529488311"
Aug 04 03:18:30.142767 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:30.142745    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Aug 04 03:18:30.843944 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:30.843909    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69dbcc4dfb-lmwkc" event={"ID":"4732185c-0a57-4163-b84b-89c007fbed25","Type":"ContainerStarted","Data":"89e470853afe38958320867ad2f4f6d2dccd321f9e4d53661f9e77488896edc0"}
Aug 04 03:18:30.860940 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:30.860888    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-69dbcc4dfb-lmwkc" podStartSLOduration=2.063538491 podStartE2EDuration="4.860872287s" totalImagesPullingTime="2.797333796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:18:30.859260176 +0000 UTC m=+541.539047317" watchObservedRunningTime="2026-08-04 03:18:30.860872287 +0000 UTC m=+541.540659406"
Aug 04 03:18:38.835103 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:18:38.835071    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-gmpnr"
Aug 04 03:19:46.843110 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:46.843077    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-685448655c-9bx5m"]
Aug 04 03:19:46.845373 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:46.845354    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:46.860415 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:46.860392    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-685448655c-9bx5m"]
Aug 04 03:19:46.915561 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:46.915531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/474ea32c-a1b4-42f5-95b6-f8221a087c74-console-oauth-config\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:46.915740 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:46.915581    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/474ea32c-a1b4-42f5-95b6-f8221a087c74-oauth-serving-cert\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:46.915740 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:46.915641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/474ea32c-a1b4-42f5-95b6-f8221a087c74-trusted-ca-bundle\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:46.915740 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:46.915678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/474ea32c-a1b4-42f5-95b6-f8221a087c74-console-serving-cert\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:46.915740 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:46.915720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/474ea32c-a1b4-42f5-95b6-f8221a087c74-console-config\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:46.915882 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:46.915762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j88xh\" (UniqueName: \"kubernetes.io/projected/474ea32c-a1b4-42f5-95b6-f8221a087c74-kube-api-access-j88xh\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:46.915882 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:46.915805    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/474ea32c-a1b4-42f5-95b6-f8221a087c74-service-ca\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:47.017198 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:47.017168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/474ea32c-a1b4-42f5-95b6-f8221a087c74-console-config\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:47.017400 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:47.017208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j88xh\" (UniqueName: \"kubernetes.io/projected/474ea32c-a1b4-42f5-95b6-f8221a087c74-kube-api-access-j88xh\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:47.017400 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:47.017226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/474ea32c-a1b4-42f5-95b6-f8221a087c74-service-ca\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:47.017400 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:47.017342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/474ea32c-a1b4-42f5-95b6-f8221a087c74-console-oauth-config\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:47.017586 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:47.017409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/474ea32c-a1b4-42f5-95b6-f8221a087c74-oauth-serving-cert\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:47.017586 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:47.017436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/474ea32c-a1b4-42f5-95b6-f8221a087c74-trusted-ca-bundle\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:47.017586 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:47.017475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/474ea32c-a1b4-42f5-95b6-f8221a087c74-console-serving-cert\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:47.018045 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:47.018015    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/474ea32c-a1b4-42f5-95b6-f8221a087c74-console-config\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:47.018155 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:47.018088    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/474ea32c-a1b4-42f5-95b6-f8221a087c74-service-ca\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:47.018224 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:47.018154    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/474ea32c-a1b4-42f5-95b6-f8221a087c74-oauth-serving-cert\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:47.018224 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:47.018202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/474ea32c-a1b4-42f5-95b6-f8221a087c74-trusted-ca-bundle\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:47.019732 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:47.019711    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/474ea32c-a1b4-42f5-95b6-f8221a087c74-console-oauth-config\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:47.019979 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:47.019939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/474ea32c-a1b4-42f5-95b6-f8221a087c74-console-serving-cert\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:47.025701 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:47.025684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j88xh\" (UniqueName: \"kubernetes.io/projected/474ea32c-a1b4-42f5-95b6-f8221a087c74-kube-api-access-j88xh\") pod \"console-685448655c-9bx5m\" (UID: \"474ea32c-a1b4-42f5-95b6-f8221a087c74\") " pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:47.154403 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:47.154314    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:47.289699 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:47.289673    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-685448655c-9bx5m"]
Aug 04 03:19:47.291944 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:19:47.291910    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474ea32c_a1b4_42f5_95b6_f8221a087c74.slice/crio-db165ace6062666cea78d28bf8e1fceb28e3046cb7eeaf2e0a26cc14dc645062 WatchSource:0}: Error finding container db165ace6062666cea78d28bf8e1fceb28e3046cb7eeaf2e0a26cc14dc645062: Status 404 returned error can't find the container with id db165ace6062666cea78d28bf8e1fceb28e3046cb7eeaf2e0a26cc14dc645062
Aug 04 03:19:48.196164 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:48.196130    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-685448655c-9bx5m" event={"ID":"474ea32c-a1b4-42f5-95b6-f8221a087c74","Type":"ContainerStarted","Data":"5ec77d9c8ed13ac72038d21992061a1a3b44b44bc3891b79f1798ea87cd68fa8"}
Aug 04 03:19:48.196626 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:48.196199    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-685448655c-9bx5m" event={"ID":"474ea32c-a1b4-42f5-95b6-f8221a087c74","Type":"ContainerStarted","Data":"db165ace6062666cea78d28bf8e1fceb28e3046cb7eeaf2e0a26cc14dc645062"}
Aug 04 03:19:48.214168 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:48.214122    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-685448655c-9bx5m" podStartSLOduration=2.214107781 podStartE2EDuration="2.214107781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:19:48.212713682 +0000 UTC m=+618.892500822" watchObservedRunningTime="2026-08-04 03:19:48.214107781 +0000 UTC m=+618.893894900"
Aug 04 03:19:57.154718 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:57.154671    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:57.155220 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:57.154747    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:57.159945 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:57.159914    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:57.240055 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:57.240029    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-685448655c-9bx5m"
Aug 04 03:19:57.294402 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:19:57.294372    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d775bd59-hd6gq"]
Aug 04 03:20:11.367707 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:11.367669    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-l6sx5"]
Aug 04 03:20:11.370001 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:11.369985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l6sx5"
Aug 04 03:20:11.372500 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:11.372478    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Aug 04 03:20:11.373030 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:11.373013    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Aug 04 03:20:11.373134 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:11.373055    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Aug 04 03:20:11.373590 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:11.373572    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Aug 04 03:20:11.373665 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:11.373579    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vjhhf\""
Aug 04 03:20:11.380055 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:11.380018    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-l6sx5"]
Aug 04 03:20:11.436208 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:11.436170    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3f95fe4-15d1-4821-a3dd-4f01792ccd30-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-l6sx5\" (UID: \"d3f95fe4-15d1-4821-a3dd-4f01792ccd30\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l6sx5"
Aug 04 03:20:11.436377 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:11.436213    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb4qk\" (UniqueName: \"kubernetes.io/projected/d3f95fe4-15d1-4821-a3dd-4f01792ccd30-kube-api-access-kb4qk\") pod \"kuadrant-console-plugin-66b5774985-l6sx5\" (UID: \"d3f95fe4-15d1-4821-a3dd-4f01792ccd30\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l6sx5"
Aug 04 03:20:11.436377 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:11.436245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d3f95fe4-15d1-4821-a3dd-4f01792ccd30-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-l6sx5\" (UID: \"d3f95fe4-15d1-4821-a3dd-4f01792ccd30\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l6sx5"
Aug 04 03:20:11.537564 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:11.537528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3f95fe4-15d1-4821-a3dd-4f01792ccd30-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-l6sx5\" (UID: \"d3f95fe4-15d1-4821-a3dd-4f01792ccd30\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l6sx5"
Aug 04 03:20:11.537564 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:11.537570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb4qk\" (UniqueName: \"kubernetes.io/projected/d3f95fe4-15d1-4821-a3dd-4f01792ccd30-kube-api-access-kb4qk\") pod \"kuadrant-console-plugin-66b5774985-l6sx5\" (UID: \"d3f95fe4-15d1-4821-a3dd-4f01792ccd30\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l6sx5"
Aug 04 03:20:11.537813 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:11.537591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d3f95fe4-15d1-4821-a3dd-4f01792ccd30-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-l6sx5\" (UID: \"d3f95fe4-15d1-4821-a3dd-4f01792ccd30\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l6sx5"
Aug 04 03:20:11.537813 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:20:11.537686    2579 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Aug 04 03:20:11.537813 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:20:11.537765    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3f95fe4-15d1-4821-a3dd-4f01792ccd30-plugin-serving-cert podName:d3f95fe4-15d1-4821-a3dd-4f01792ccd30 nodeName:}" failed. No retries permitted until 2026-08-04 03:20:12.03774781 +0000 UTC m=+642.717534907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/d3f95fe4-15d1-4821-a3dd-4f01792ccd30-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-l6sx5" (UID: "d3f95fe4-15d1-4821-a3dd-4f01792ccd30") : secret "plugin-serving-cert" not found
Aug 04 03:20:11.538218 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:11.538201    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d3f95fe4-15d1-4821-a3dd-4f01792ccd30-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-l6sx5\" (UID: \"d3f95fe4-15d1-4821-a3dd-4f01792ccd30\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l6sx5"
Aug 04 03:20:11.550427 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:11.550404    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb4qk\" (UniqueName: \"kubernetes.io/projected/d3f95fe4-15d1-4821-a3dd-4f01792ccd30-kube-api-access-kb4qk\") pod \"kuadrant-console-plugin-66b5774985-l6sx5\" (UID: \"d3f95fe4-15d1-4821-a3dd-4f01792ccd30\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l6sx5"
Aug 04 03:20:12.042187 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:12.042149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3f95fe4-15d1-4821-a3dd-4f01792ccd30-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-l6sx5\" (UID: \"d3f95fe4-15d1-4821-a3dd-4f01792ccd30\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l6sx5"
Aug 04 03:20:12.044618 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:12.044596    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3f95fe4-15d1-4821-a3dd-4f01792ccd30-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-l6sx5\" (UID: \"d3f95fe4-15d1-4821-a3dd-4f01792ccd30\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l6sx5"
Aug 04 03:20:12.282006 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:12.281942    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l6sx5"
Aug 04 03:20:12.410091 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:12.410066    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-l6sx5"]
Aug 04 03:20:12.415516 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:20:12.415484    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3f95fe4_15d1_4821_a3dd_4f01792ccd30.slice/crio-00c87ee6c4086450d9cb4e753c5dbcd98c5cc0948c970a9e16ca809d8adb3747 WatchSource:0}: Error finding container 00c87ee6c4086450d9cb4e753c5dbcd98c5cc0948c970a9e16ca809d8adb3747: Status 404 returned error can't find the container with id 00c87ee6c4086450d9cb4e753c5dbcd98c5cc0948c970a9e16ca809d8adb3747
Aug 04 03:20:12.417126 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:12.417106    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 03:20:13.307324 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:13.307278    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l6sx5" event={"ID":"d3f95fe4-15d1-4821-a3dd-4f01792ccd30","Type":"ContainerStarted","Data":"00c87ee6c4086450d9cb4e753c5dbcd98c5cc0948c970a9e16ca809d8adb3747"}
Aug 04 03:20:22.315126 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:22.315055    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d775bd59-hd6gq" podUID="32cadeb2-cb8a-46e5-9f33-fde888a05973" containerName="console" containerID="cri-o://5ed7034d20958858dbeb3d9617ceba377dd2c19d4520375ec268df42c5766950" gracePeriod=15
Aug 04 03:20:29.261718 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:29.261682    2579 patch_prober.go:28] interesting pod/console-5d775bd59-hd6gq container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.132.0.27:8443/health\": dial tcp 10.132.0.27:8443: connect: connection refused" start-of-body=
Aug 04 03:20:29.262367 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:29.261747    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-5d775bd59-hd6gq" podUID="32cadeb2-cb8a-46e5-9f33-fde888a05973" containerName="console" probeResult="failure" output="Get \"https://10.132.0.27:8443/health\": dial tcp 10.132.0.27:8443: connect: connection refused"
Aug 04 03:20:34.021514 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.021479    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:20:34.155406 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.155314    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-console-config\") pod \"32cadeb2-cb8a-46e5-9f33-fde888a05973\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") "
Aug 04 03:20:34.155406 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.155358    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32cadeb2-cb8a-46e5-9f33-fde888a05973-console-oauth-config\") pod \"32cadeb2-cb8a-46e5-9f33-fde888a05973\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") "
Aug 04 03:20:34.155406 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.155385    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb72s\" (UniqueName: \"kubernetes.io/projected/32cadeb2-cb8a-46e5-9f33-fde888a05973-kube-api-access-vb72s\") pod \"32cadeb2-cb8a-46e5-9f33-fde888a05973\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") "
Aug 04 03:20:34.155684 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.155433    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-service-ca\") pod \"32cadeb2-cb8a-46e5-9f33-fde888a05973\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") "
Aug 04 03:20:34.155684 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.155474    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-oauth-serving-cert\") pod \"32cadeb2-cb8a-46e5-9f33-fde888a05973\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") "
Aug 04 03:20:34.155684 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.155517    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32cadeb2-cb8a-46e5-9f33-fde888a05973-console-serving-cert\") pod \"32cadeb2-cb8a-46e5-9f33-fde888a05973\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") "
Aug 04 03:20:34.155684 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.155544    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-trusted-ca-bundle\") pod \"32cadeb2-cb8a-46e5-9f33-fde888a05973\" (UID: \"32cadeb2-cb8a-46e5-9f33-fde888a05973\") "
Aug 04 03:20:34.155859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.155766    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-console-config" (OuterVolumeSpecName: "console-config") pod "32cadeb2-cb8a-46e5-9f33-fde888a05973" (UID: "32cadeb2-cb8a-46e5-9f33-fde888a05973"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:20:34.155904 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.155890    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-console-config\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:20:34.155939 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.155901    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-service-ca" (OuterVolumeSpecName: "service-ca") pod "32cadeb2-cb8a-46e5-9f33-fde888a05973" (UID: "32cadeb2-cb8a-46e5-9f33-fde888a05973"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:20:34.156190 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.156162    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "32cadeb2-cb8a-46e5-9f33-fde888a05973" (UID: "32cadeb2-cb8a-46e5-9f33-fde888a05973"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:20:34.156383 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.156363    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "32cadeb2-cb8a-46e5-9f33-fde888a05973" (UID: "32cadeb2-cb8a-46e5-9f33-fde888a05973"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:20:34.157634 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.157605    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cadeb2-cb8a-46e5-9f33-fde888a05973-kube-api-access-vb72s" (OuterVolumeSpecName: "kube-api-access-vb72s") pod "32cadeb2-cb8a-46e5-9f33-fde888a05973" (UID: "32cadeb2-cb8a-46e5-9f33-fde888a05973"). InnerVolumeSpecName "kube-api-access-vb72s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:20:34.157718 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.157634    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32cadeb2-cb8a-46e5-9f33-fde888a05973-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "32cadeb2-cb8a-46e5-9f33-fde888a05973" (UID: "32cadeb2-cb8a-46e5-9f33-fde888a05973"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:20:34.158014 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.157993    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32cadeb2-cb8a-46e5-9f33-fde888a05973-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "32cadeb2-cb8a-46e5-9f33-fde888a05973" (UID: "32cadeb2-cb8a-46e5-9f33-fde888a05973"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:20:34.257333 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.257294    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32cadeb2-cb8a-46e5-9f33-fde888a05973-console-oauth-config\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:20:34.257333 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.257325    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb72s\" (UniqueName: \"kubernetes.io/projected/32cadeb2-cb8a-46e5-9f33-fde888a05973-kube-api-access-vb72s\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:20:34.257333 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.257336    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:20:34.257333 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.257346    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-oauth-serving-cert\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:20:34.257595 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.257355    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32cadeb2-cb8a-46e5-9f33-fde888a05973-console-serving-cert\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:20:34.257595 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.257365    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32cadeb2-cb8a-46e5-9f33-fde888a05973-trusted-ca-bundle\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:20:34.427190 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.427119    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l6sx5" event={"ID":"d3f95fe4-15d1-4821-a3dd-4f01792ccd30","Type":"ContainerStarted","Data":"0b7ce9fa969a0d79b347260695964382d1411204d3d12f775e63d48d66d0bd4c"}
Aug 04 03:20:34.428367 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.428344    2579 generic.go:356] "Generic (PLEG): container finished" podID="32cadeb2-cb8a-46e5-9f33-fde888a05973" containerID="5ed7034d20958858dbeb3d9617ceba377dd2c19d4520375ec268df42c5766950" exitCode=0
Aug 04 03:20:34.428466 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.428382    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d775bd59-hd6gq"
Aug 04 03:20:34.428466 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.428411    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d775bd59-hd6gq" event={"ID":"32cadeb2-cb8a-46e5-9f33-fde888a05973","Type":"ContainerDied","Data":"5ed7034d20958858dbeb3d9617ceba377dd2c19d4520375ec268df42c5766950"}
Aug 04 03:20:34.428466 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.428443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d775bd59-hd6gq" event={"ID":"32cadeb2-cb8a-46e5-9f33-fde888a05973","Type":"ContainerDied","Data":"e09e9a51104596b212d8bf8949ba878d089d665e5fd72454dd917c9d7b9d48fb"}
Aug 04 03:20:34.428466 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.428462    2579 scope.go:117] "RemoveContainer" containerID="5ed7034d20958858dbeb3d9617ceba377dd2c19d4520375ec268df42c5766950"
Aug 04 03:20:34.437533 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.437508    2579 scope.go:117] "RemoveContainer" containerID="5ed7034d20958858dbeb3d9617ceba377dd2c19d4520375ec268df42c5766950"
Aug 04 03:20:34.437838 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:20:34.437819    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed7034d20958858dbeb3d9617ceba377dd2c19d4520375ec268df42c5766950\": container with ID starting with 5ed7034d20958858dbeb3d9617ceba377dd2c19d4520375ec268df42c5766950 not found: ID does not exist" containerID="5ed7034d20958858dbeb3d9617ceba377dd2c19d4520375ec268df42c5766950"
Aug 04 03:20:34.437902 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.437848    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed7034d20958858dbeb3d9617ceba377dd2c19d4520375ec268df42c5766950"} err="failed to get container status \"5ed7034d20958858dbeb3d9617ceba377dd2c19d4520375ec268df42c5766950\": rpc error: code = NotFound desc = could not find container \"5ed7034d20958858dbeb3d9617ceba377dd2c19d4520375ec268df42c5766950\": container with ID starting with 5ed7034d20958858dbeb3d9617ceba377dd2c19d4520375ec268df42c5766950 not found: ID does not exist"
Aug 04 03:20:34.444650 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.444603    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l6sx5" podStartSLOduration=1.8967516199999999 podStartE2EDuration="23.444587628s" totalImagesPullingTime="21.547836008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:20:34.443347615 +0000 UTC m=+665.123134735" watchObservedRunningTime="2026-08-04 03:20:34.444587628 +0000 UTC m=+665.124374747"
Aug 04 03:20:34.460485 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.460453    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d775bd59-hd6gq"]
Aug 04 03:20:34.466484 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:34.466456    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d775bd59-hd6gq"]
Aug 04 03:20:35.845222 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:35.845190    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32cadeb2-cb8a-46e5-9f33-fde888a05973" path="/var/lib/kubelet/pods/32cadeb2-cb8a-46e5-9f33-fde888a05973/volumes"
Aug 04 03:20:52.609750 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:52.609717    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-smftw"]
Aug 04 03:20:52.610248 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:52.610121    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32cadeb2-cb8a-46e5-9f33-fde888a05973" containerName="console"
Aug 04 03:20:52.610248 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:52.610134    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="32cadeb2-cb8a-46e5-9f33-fde888a05973" containerName="console"
Aug 04 03:20:52.610248 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:52.610213    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="32cadeb2-cb8a-46e5-9f33-fde888a05973" containerName="console"
Aug 04 03:20:52.669941 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:52.669909    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-smftw"]
Aug 04 03:20:52.670133 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:52.670038    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-smftw"
Aug 04 03:20:52.672490 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:52.672469    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Aug 04 03:20:52.709015 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:52.708983    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-smftw"]
Aug 04 03:20:52.821099 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:52.821065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l47hf\" (UniqueName: \"kubernetes.io/projected/ee40ab07-4527-4e66-adf5-e060cfd81e26-kube-api-access-l47hf\") pod \"limitador-limitador-64c94fd64f-smftw\" (UID: \"ee40ab07-4527-4e66-adf5-e060cfd81e26\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-smftw"
Aug 04 03:20:52.821298 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:52.821187    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ee40ab07-4527-4e66-adf5-e060cfd81e26-config-file\") pod \"limitador-limitador-64c94fd64f-smftw\" (UID: \"ee40ab07-4527-4e66-adf5-e060cfd81e26\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-smftw"
Aug 04 03:20:52.922095 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:52.922007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ee40ab07-4527-4e66-adf5-e060cfd81e26-config-file\") pod \"limitador-limitador-64c94fd64f-smftw\" (UID: \"ee40ab07-4527-4e66-adf5-e060cfd81e26\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-smftw"
Aug 04 03:20:52.922095 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:52.922050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l47hf\" (UniqueName: \"kubernetes.io/projected/ee40ab07-4527-4e66-adf5-e060cfd81e26-kube-api-access-l47hf\") pod \"limitador-limitador-64c94fd64f-smftw\" (UID: \"ee40ab07-4527-4e66-adf5-e060cfd81e26\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-smftw"
Aug 04 03:20:52.922692 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:52.922665    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ee40ab07-4527-4e66-adf5-e060cfd81e26-config-file\") pod \"limitador-limitador-64c94fd64f-smftw\" (UID: \"ee40ab07-4527-4e66-adf5-e060cfd81e26\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-smftw"
Aug 04 03:20:52.930623 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:52.930598    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l47hf\" (UniqueName: \"kubernetes.io/projected/ee40ab07-4527-4e66-adf5-e060cfd81e26-kube-api-access-l47hf\") pod \"limitador-limitador-64c94fd64f-smftw\" (UID: \"ee40ab07-4527-4e66-adf5-e060cfd81e26\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-smftw"
Aug 04 03:20:52.981809 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:52.981782    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-smftw"
Aug 04 03:20:53.113448 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:53.113424    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-smftw"]
Aug 04 03:20:53.115110 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:20:53.115084    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee40ab07_4527_4e66_adf5_e060cfd81e26.slice/crio-81efc62547520dc5f93ad4e172aaa86114efcb0da7bf542f0140925104f836e7 WatchSource:0}: Error finding container 81efc62547520dc5f93ad4e172aaa86114efcb0da7bf542f0140925104f836e7: Status 404 returned error can't find the container with id 81efc62547520dc5f93ad4e172aaa86114efcb0da7bf542f0140925104f836e7
Aug 04 03:20:53.521461 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:53.521427    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-smftw" event={"ID":"ee40ab07-4527-4e66-adf5-e060cfd81e26","Type":"ContainerStarted","Data":"81efc62547520dc5f93ad4e172aaa86114efcb0da7bf542f0140925104f836e7"}
Aug 04 03:20:56.540449 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:56.540417    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-smftw" event={"ID":"ee40ab07-4527-4e66-adf5-e060cfd81e26","Type":"ContainerStarted","Data":"b5eeb1dd7035af9552d2997f685fb5891d873b3440751e9d5f76ed020d938cea"}
Aug 04 03:20:56.540839 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:56.540551    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-smftw"
Aug 04 03:20:56.556972 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:20:56.556910    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-smftw" podStartSLOduration=1.631627911 podStartE2EDuration="4.556895548s" totalImagesPullingTime="2.925267637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:20:56.556331694 +0000 UTC m=+687.236118847" watchObservedRunningTime="2026-08-04 03:20:56.556895548 +0000 UTC m=+687.236682666"
Aug 04 03:21:07.254037 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:07.254001    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-smftw"]
Aug 04 03:21:07.254511 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:07.254196    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-smftw" podUID="ee40ab07-4527-4e66-adf5-e060cfd81e26" containerName="limitador" containerID="cri-o://b5eeb1dd7035af9552d2997f685fb5891d873b3440751e9d5f76ed020d938cea" gracePeriod=30
Aug 04 03:21:07.254906 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:07.254889    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-smftw"
Aug 04 03:21:07.825955 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:07.825890    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-smftw"
Aug 04 03:21:07.856760 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:07.856733    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l47hf\" (UniqueName: \"kubernetes.io/projected/ee40ab07-4527-4e66-adf5-e060cfd81e26-kube-api-access-l47hf\") pod \"ee40ab07-4527-4e66-adf5-e060cfd81e26\" (UID: \"ee40ab07-4527-4e66-adf5-e060cfd81e26\") "
Aug 04 03:21:07.856916 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:07.856780    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ee40ab07-4527-4e66-adf5-e060cfd81e26-config-file\") pod \"ee40ab07-4527-4e66-adf5-e060cfd81e26\" (UID: \"ee40ab07-4527-4e66-adf5-e060cfd81e26\") "
Aug 04 03:21:07.857154 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:07.857134    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee40ab07-4527-4e66-adf5-e060cfd81e26-config-file" (OuterVolumeSpecName: "config-file") pod "ee40ab07-4527-4e66-adf5-e060cfd81e26" (UID: "ee40ab07-4527-4e66-adf5-e060cfd81e26"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:21:07.858947 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:07.858927    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee40ab07-4527-4e66-adf5-e060cfd81e26-kube-api-access-l47hf" (OuterVolumeSpecName: "kube-api-access-l47hf") pod "ee40ab07-4527-4e66-adf5-e060cfd81e26" (UID: "ee40ab07-4527-4e66-adf5-e060cfd81e26"). InnerVolumeSpecName "kube-api-access-l47hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:21:07.958364 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:07.958310    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l47hf\" (UniqueName: \"kubernetes.io/projected/ee40ab07-4527-4e66-adf5-e060cfd81e26-kube-api-access-l47hf\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:21:07.958364 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:07.958353    2579 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ee40ab07-4527-4e66-adf5-e060cfd81e26-config-file\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:21:08.598871 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:08.598840    2579 generic.go:356] "Generic (PLEG): container finished" podID="ee40ab07-4527-4e66-adf5-e060cfd81e26" containerID="b5eeb1dd7035af9552d2997f685fb5891d873b3440751e9d5f76ed020d938cea" exitCode=0
Aug 04 03:21:08.599305 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:08.598884    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-smftw"
Aug 04 03:21:08.599305 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:08.598930    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-smftw" event={"ID":"ee40ab07-4527-4e66-adf5-e060cfd81e26","Type":"ContainerDied","Data":"b5eeb1dd7035af9552d2997f685fb5891d873b3440751e9d5f76ed020d938cea"}
Aug 04 03:21:08.599305 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:08.598989    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-smftw" event={"ID":"ee40ab07-4527-4e66-adf5-e060cfd81e26","Type":"ContainerDied","Data":"81efc62547520dc5f93ad4e172aaa86114efcb0da7bf542f0140925104f836e7"}
Aug 04 03:21:08.599305 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:08.599013    2579 scope.go:117] "RemoveContainer" containerID="b5eeb1dd7035af9552d2997f685fb5891d873b3440751e9d5f76ed020d938cea"
Aug 04 03:21:08.607821 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:08.607798    2579 scope.go:117] "RemoveContainer" containerID="b5eeb1dd7035af9552d2997f685fb5891d873b3440751e9d5f76ed020d938cea"
Aug 04 03:21:08.608123 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:21:08.608102    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5eeb1dd7035af9552d2997f685fb5891d873b3440751e9d5f76ed020d938cea\": container with ID starting with b5eeb1dd7035af9552d2997f685fb5891d873b3440751e9d5f76ed020d938cea not found: ID does not exist" containerID="b5eeb1dd7035af9552d2997f685fb5891d873b3440751e9d5f76ed020d938cea"
Aug 04 03:21:08.608200 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:08.608135    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5eeb1dd7035af9552d2997f685fb5891d873b3440751e9d5f76ed020d938cea"} err="failed to get container status \"b5eeb1dd7035af9552d2997f685fb5891d873b3440751e9d5f76ed020d938cea\": rpc error: code = NotFound desc = could not find container \"b5eeb1dd7035af9552d2997f685fb5891d873b3440751e9d5f76ed020d938cea\": container with ID starting with b5eeb1dd7035af9552d2997f685fb5891d873b3440751e9d5f76ed020d938cea not found: ID does not exist"
Aug 04 03:21:08.619516 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:08.619489    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-smftw"]
Aug 04 03:21:08.623314 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:08.623292    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-smftw"]
Aug 04 03:21:09.114029 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.107393    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-twdj2"]
Aug 04 03:21:09.114029 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.108507    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee40ab07-4527-4e66-adf5-e060cfd81e26" containerName="limitador"
Aug 04 03:21:09.114029 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.108534    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee40ab07-4527-4e66-adf5-e060cfd81e26" containerName="limitador"
Aug 04 03:21:09.114029 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.108712    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee40ab07-4527-4e66-adf5-e060cfd81e26" containerName="limitador"
Aug 04 03:21:09.114029 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.113729    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-twdj2"
Aug 04 03:21:09.116566 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.116539    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-2z68n\""
Aug 04 03:21:09.116716 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.116546    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Aug 04 03:21:09.116926 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.116904    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Aug 04 03:21:09.117062 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.116944    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Aug 04 03:21:09.118068 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.118045    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-twdj2"]
Aug 04 03:21:09.168686 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.168643    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0caa0967-9ea2-4918-a068-d06269400223-data\") pod \"postgres-69fb4594cf-twdj2\" (UID: \"0caa0967-9ea2-4918-a068-d06269400223\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-twdj2"
Aug 04 03:21:09.168686 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.168683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvqh8\" (UniqueName: \"kubernetes.io/projected/0caa0967-9ea2-4918-a068-d06269400223-kube-api-access-xvqh8\") pod \"postgres-69fb4594cf-twdj2\" (UID: \"0caa0967-9ea2-4918-a068-d06269400223\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-twdj2"
Aug 04 03:21:09.270279 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.270238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0caa0967-9ea2-4918-a068-d06269400223-data\") pod \"postgres-69fb4594cf-twdj2\" (UID: \"0caa0967-9ea2-4918-a068-d06269400223\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-twdj2"
Aug 04 03:21:09.270279 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.270278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvqh8\" (UniqueName: \"kubernetes.io/projected/0caa0967-9ea2-4918-a068-d06269400223-kube-api-access-xvqh8\") pod \"postgres-69fb4594cf-twdj2\" (UID: \"0caa0967-9ea2-4918-a068-d06269400223\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-twdj2"
Aug 04 03:21:09.270644 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.270622    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0caa0967-9ea2-4918-a068-d06269400223-data\") pod \"postgres-69fb4594cf-twdj2\" (UID: \"0caa0967-9ea2-4918-a068-d06269400223\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-twdj2"
Aug 04 03:21:09.278349 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.278318    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvqh8\" (UniqueName: \"kubernetes.io/projected/0caa0967-9ea2-4918-a068-d06269400223-kube-api-access-xvqh8\") pod \"postgres-69fb4594cf-twdj2\" (UID: \"0caa0967-9ea2-4918-a068-d06269400223\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-twdj2"
Aug 04 03:21:09.424948 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.424871    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-twdj2"
Aug 04 03:21:09.562984 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.562935    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-twdj2"]
Aug 04 03:21:09.565098 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:21:09.565072    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0caa0967_9ea2_4918_a068_d06269400223.slice/crio-62a0ae7e34b7920d87e86deb7c1ce5d564dcfccb4ec3295f4ada32454f5c7bd5 WatchSource:0}: Error finding container 62a0ae7e34b7920d87e86deb7c1ce5d564dcfccb4ec3295f4ada32454f5c7bd5: Status 404 returned error can't find the container with id 62a0ae7e34b7920d87e86deb7c1ce5d564dcfccb4ec3295f4ada32454f5c7bd5
Aug 04 03:21:09.605351 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.605316    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-twdj2" event={"ID":"0caa0967-9ea2-4918-a068-d06269400223","Type":"ContainerStarted","Data":"62a0ae7e34b7920d87e86deb7c1ce5d564dcfccb4ec3295f4ada32454f5c7bd5"}
Aug 04 03:21:09.844859 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:09.844828    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee40ab07-4527-4e66-adf5-e060cfd81e26" path="/var/lib/kubelet/pods/ee40ab07-4527-4e66-adf5-e060cfd81e26/volumes"
Aug 04 03:21:14.634107 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:14.634074    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-twdj2" event={"ID":"0caa0967-9ea2-4918-a068-d06269400223","Type":"ContainerStarted","Data":"6c6d1e854fe5b109ec568e25a3db78d59f2441ea5f53efd5aca5d42f3ad36061"}
Aug 04 03:21:14.634555 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:14.634186    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-twdj2"
Aug 04 03:21:14.650017 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:14.649953    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-twdj2" podStartSLOduration=0.719473782 podStartE2EDuration="5.649936473s" totalImagesPullingTime="4.930462691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:21:14.648267488 +0000 UTC m=+705.328054608" watchObservedRunningTime="2026-08-04 03:21:14.649936473 +0000 UTC m=+705.329723591"
Aug 04 03:21:20.667606 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:20.667576    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-twdj2"
Aug 04 03:21:27.253564 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.253481    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d6db9d6fd-c4vlg"]
Aug 04 03:21:27.261687 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.261663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d6db9d6fd-c4vlg"
Aug 04 03:21:27.264332 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.264311    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Aug 04 03:21:27.264807 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.264790    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Aug 04 03:21:27.265271 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.265255    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7jvk5\""
Aug 04 03:21:27.272116 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.272096    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d6db9d6fd-c4vlg"]
Aug 04 03:21:27.335237 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.335198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t5q2\" (UniqueName: \"kubernetes.io/projected/e334ec4d-8a08-4d74-9a70-5b572b659b78-kube-api-access-8t5q2\") pod \"maas-controller-d6db9d6fd-c4vlg\" (UID: \"e334ec4d-8a08-4d74-9a70-5b572b659b78\") " pod="opendatahub/maas-controller-d6db9d6fd-c4vlg"
Aug 04 03:21:27.335410 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.335273    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e334ec4d-8a08-4d74-9a70-5b572b659b78-webhook-cert\") pod \"maas-controller-d6db9d6fd-c4vlg\" (UID: \"e334ec4d-8a08-4d74-9a70-5b572b659b78\") " pod="opendatahub/maas-controller-d6db9d6fd-c4vlg"
Aug 04 03:21:27.412374 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.412341    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5465dcb4d8-w2ksw"]
Aug 04 03:21:27.418661 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.418635    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5465dcb4d8-w2ksw"
Aug 04 03:21:27.425323 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.425295    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5465dcb4d8-w2ksw"]
Aug 04 03:21:27.436493 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.436456    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t5q2\" (UniqueName: \"kubernetes.io/projected/e334ec4d-8a08-4d74-9a70-5b572b659b78-kube-api-access-8t5q2\") pod \"maas-controller-d6db9d6fd-c4vlg\" (UID: \"e334ec4d-8a08-4d74-9a70-5b572b659b78\") " pod="opendatahub/maas-controller-d6db9d6fd-c4vlg"
Aug 04 03:21:27.436673 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.436523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e334ec4d-8a08-4d74-9a70-5b572b659b78-webhook-cert\") pod \"maas-controller-d6db9d6fd-c4vlg\" (UID: \"e334ec4d-8a08-4d74-9a70-5b572b659b78\") " pod="opendatahub/maas-controller-d6db9d6fd-c4vlg"
Aug 04 03:21:27.436761 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:21:27.436709    2579 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Aug 04 03:21:27.436818 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:21:27.436784    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e334ec4d-8a08-4d74-9a70-5b572b659b78-webhook-cert podName:e334ec4d-8a08-4d74-9a70-5b572b659b78 nodeName:}" failed. No retries permitted until 2026-08-04 03:21:27.936764777 +0000 UTC m=+718.616551874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/e334ec4d-8a08-4d74-9a70-5b572b659b78-webhook-cert") pod "maas-controller-d6db9d6fd-c4vlg" (UID: "e334ec4d-8a08-4d74-9a70-5b572b659b78") : secret "maas-controller-webhook-cert" not found
Aug 04 03:21:27.444974 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.444939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t5q2\" (UniqueName: \"kubernetes.io/projected/e334ec4d-8a08-4d74-9a70-5b572b659b78-kube-api-access-8t5q2\") pod \"maas-controller-d6db9d6fd-c4vlg\" (UID: \"e334ec4d-8a08-4d74-9a70-5b572b659b78\") " pod="opendatahub/maas-controller-d6db9d6fd-c4vlg"
Aug 04 03:21:27.537473 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.537390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d-webhook-cert\") pod \"maas-controller-5465dcb4d8-w2ksw\" (UID: \"0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d\") " pod="opendatahub/maas-controller-5465dcb4d8-w2ksw"
Aug 04 03:21:27.537473 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.537453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpx6q\" (UniqueName: \"kubernetes.io/projected/0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d-kube-api-access-lpx6q\") pod \"maas-controller-5465dcb4d8-w2ksw\" (UID: \"0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d\") " pod="opendatahub/maas-controller-5465dcb4d8-w2ksw"
Aug 04 03:21:27.638475 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.638440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d-webhook-cert\") pod \"maas-controller-5465dcb4d8-w2ksw\" (UID: \"0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d\") " pod="opendatahub/maas-controller-5465dcb4d8-w2ksw"
Aug 04 03:21:27.638679 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.638491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpx6q\" (UniqueName: \"kubernetes.io/projected/0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d-kube-api-access-lpx6q\") pod \"maas-controller-5465dcb4d8-w2ksw\" (UID: \"0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d\") " pod="opendatahub/maas-controller-5465dcb4d8-w2ksw"
Aug 04 03:21:27.638679 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:21:27.638615    2579 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Aug 04 03:21:27.638799 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:21:27.638704    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d-webhook-cert podName:0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d nodeName:}" failed. No retries permitted until 2026-08-04 03:21:28.138683182 +0000 UTC m=+718.818470288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d-webhook-cert") pod "maas-controller-5465dcb4d8-w2ksw" (UID: "0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d") : secret "maas-controller-webhook-cert" not found
Aug 04 03:21:27.646661 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.646635    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpx6q\" (UniqueName: \"kubernetes.io/projected/0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d-kube-api-access-lpx6q\") pod \"maas-controller-5465dcb4d8-w2ksw\" (UID: \"0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d\") " pod="opendatahub/maas-controller-5465dcb4d8-w2ksw"
Aug 04 03:21:27.940318 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.940218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e334ec4d-8a08-4d74-9a70-5b572b659b78-webhook-cert\") pod \"maas-controller-d6db9d6fd-c4vlg\" (UID: \"e334ec4d-8a08-4d74-9a70-5b572b659b78\") " pod="opendatahub/maas-controller-d6db9d6fd-c4vlg"
Aug 04 03:21:27.942576 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:27.942547    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e334ec4d-8a08-4d74-9a70-5b572b659b78-webhook-cert\") pod \"maas-controller-d6db9d6fd-c4vlg\" (UID: \"e334ec4d-8a08-4d74-9a70-5b572b659b78\") " pod="opendatahub/maas-controller-d6db9d6fd-c4vlg"
Aug 04 03:21:28.142864 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:28.142828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d-webhook-cert\") pod \"maas-controller-5465dcb4d8-w2ksw\" (UID: \"0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d\") " pod="opendatahub/maas-controller-5465dcb4d8-w2ksw"
Aug 04 03:21:28.145359 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:28.145334    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d-webhook-cert\") pod \"maas-controller-5465dcb4d8-w2ksw\" (UID: \"0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d\") " pod="opendatahub/maas-controller-5465dcb4d8-w2ksw"
Aug 04 03:21:28.174249 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:28.174214    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d6db9d6fd-c4vlg"
Aug 04 03:21:28.307131 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:28.307106    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d6db9d6fd-c4vlg"]
Aug 04 03:21:28.309677 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:21:28.309651    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode334ec4d_8a08_4d74_9a70_5b572b659b78.slice/crio-2715ccf83856ce0b0058c30880d9346a241e24a60817d30eaf5bbe40d4e307b4 WatchSource:0}: Error finding container 2715ccf83856ce0b0058c30880d9346a241e24a60817d30eaf5bbe40d4e307b4: Status 404 returned error can't find the container with id 2715ccf83856ce0b0058c30880d9346a241e24a60817d30eaf5bbe40d4e307b4
Aug 04 03:21:28.331425 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:28.331396    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5465dcb4d8-w2ksw"
Aug 04 03:21:28.460275 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:28.460253    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5465dcb4d8-w2ksw"]
Aug 04 03:21:28.462427 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:21:28.462396    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4530c0_c12b_4a6e_b76d_f5fa98b4a17d.slice/crio-e3ac2c1f5b4509d91bcbfc774f0df85389a94a85be52fde6fdbc32b2259a025f WatchSource:0}: Error finding container e3ac2c1f5b4509d91bcbfc774f0df85389a94a85be52fde6fdbc32b2259a025f: Status 404 returned error can't find the container with id e3ac2c1f5b4509d91bcbfc774f0df85389a94a85be52fde6fdbc32b2259a025f
Aug 04 03:21:28.701737 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:28.701651    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d6db9d6fd-c4vlg" event={"ID":"e334ec4d-8a08-4d74-9a70-5b572b659b78","Type":"ContainerStarted","Data":"2715ccf83856ce0b0058c30880d9346a241e24a60817d30eaf5bbe40d4e307b4"}
Aug 04 03:21:28.702749 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:28.702728    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5465dcb4d8-w2ksw" event={"ID":"0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d","Type":"ContainerStarted","Data":"e3ac2c1f5b4509d91bcbfc774f0df85389a94a85be52fde6fdbc32b2259a025f"}
Aug 04 03:21:30.848157 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:30.848133    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Aug 04 03:21:31.720916 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:31.720881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5465dcb4d8-w2ksw" event={"ID":"0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d","Type":"ContainerStarted","Data":"49b8b9785a644caa439eab78b33210ec1f8883156e8d703cb40d62d91e89aded"}
Aug 04 03:21:31.721122 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:31.721056    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5465dcb4d8-w2ksw"
Aug 04 03:21:31.722341 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:31.722317    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d6db9d6fd-c4vlg" event={"ID":"e334ec4d-8a08-4d74-9a70-5b572b659b78","Type":"ContainerStarted","Data":"642931b6a9b0a0a6357aa2b0425a11c854ad8b40d9d8b444ab87abcd6aeaa82e"}
Aug 04 03:21:31.722461 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:31.722442    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d6db9d6fd-c4vlg"
Aug 04 03:21:31.743028 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:31.742945    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5465dcb4d8-w2ksw" podStartSLOduration=2.35492818 podStartE2EDuration="4.742929209s" totalImagesPullingTime="2.388001029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:21:31.740821913 +0000 UTC m=+722.420609047" watchObservedRunningTime="2026-08-04 03:21:31.742929209 +0000 UTC m=+722.422716328"
Aug 04 03:21:31.757941 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:31.757889    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d6db9d6fd-c4vlg" podStartSLOduration=2.223276791 podStartE2EDuration="4.75787237s" totalImagesPullingTime="2.534595579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:21:31.756834125 +0000 UTC m=+722.436621269" watchObservedRunningTime="2026-08-04 03:21:31.75787237 +0000 UTC m=+722.437659488"
Aug 04 03:21:42.729825 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:42.729796    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5465dcb4d8-w2ksw"
Aug 04 03:21:42.730243 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:42.729876    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d6db9d6fd-c4vlg"
Aug 04 03:21:42.794391 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:42.794341    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d6db9d6fd-c4vlg"]
Aug 04 03:21:42.794654 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:42.794614    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d6db9d6fd-c4vlg" podUID="e334ec4d-8a08-4d74-9a70-5b572b659b78" containerName="manager" containerID="cri-o://642931b6a9b0a0a6357aa2b0425a11c854ad8b40d9d8b444ab87abcd6aeaa82e" gracePeriod=30
Aug 04 03:21:43.056465 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:43.056436    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d6db9d6fd-c4vlg"
Aug 04 03:21:43.080050 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:43.080020    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t5q2\" (UniqueName: \"kubernetes.io/projected/e334ec4d-8a08-4d74-9a70-5b572b659b78-kube-api-access-8t5q2\") pod \"e334ec4d-8a08-4d74-9a70-5b572b659b78\" (UID: \"e334ec4d-8a08-4d74-9a70-5b572b659b78\") "
Aug 04 03:21:43.080228 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:43.080070    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e334ec4d-8a08-4d74-9a70-5b572b659b78-webhook-cert\") pod \"e334ec4d-8a08-4d74-9a70-5b572b659b78\" (UID: \"e334ec4d-8a08-4d74-9a70-5b572b659b78\") "
Aug 04 03:21:43.082403 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:43.082359    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e334ec4d-8a08-4d74-9a70-5b572b659b78-kube-api-access-8t5q2" (OuterVolumeSpecName: "kube-api-access-8t5q2") pod "e334ec4d-8a08-4d74-9a70-5b572b659b78" (UID: "e334ec4d-8a08-4d74-9a70-5b572b659b78"). InnerVolumeSpecName "kube-api-access-8t5q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:21:43.082533 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:43.082481    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e334ec4d-8a08-4d74-9a70-5b572b659b78-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e334ec4d-8a08-4d74-9a70-5b572b659b78" (UID: "e334ec4d-8a08-4d74-9a70-5b572b659b78"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:21:43.181180 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:43.181143    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t5q2\" (UniqueName: \"kubernetes.io/projected/e334ec4d-8a08-4d74-9a70-5b572b659b78-kube-api-access-8t5q2\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:21:43.181180 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:43.181175    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e334ec4d-8a08-4d74-9a70-5b572b659b78-webhook-cert\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:21:43.778982 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:43.778931    2579 generic.go:356] "Generic (PLEG): container finished" podID="e334ec4d-8a08-4d74-9a70-5b572b659b78" containerID="642931b6a9b0a0a6357aa2b0425a11c854ad8b40d9d8b444ab87abcd6aeaa82e" exitCode=0
Aug 04 03:21:43.779382 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:43.779005    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d6db9d6fd-c4vlg"
Aug 04 03:21:43.779382 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:43.779022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d6db9d6fd-c4vlg" event={"ID":"e334ec4d-8a08-4d74-9a70-5b572b659b78","Type":"ContainerDied","Data":"642931b6a9b0a0a6357aa2b0425a11c854ad8b40d9d8b444ab87abcd6aeaa82e"}
Aug 04 03:21:43.779382 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:43.779116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d6db9d6fd-c4vlg" event={"ID":"e334ec4d-8a08-4d74-9a70-5b572b659b78","Type":"ContainerDied","Data":"2715ccf83856ce0b0058c30880d9346a241e24a60817d30eaf5bbe40d4e307b4"}
Aug 04 03:21:43.779382 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:43.779146    2579 scope.go:117] "RemoveContainer" containerID="642931b6a9b0a0a6357aa2b0425a11c854ad8b40d9d8b444ab87abcd6aeaa82e"
Aug 04 03:21:43.788259 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:43.788243    2579 scope.go:117] "RemoveContainer" containerID="642931b6a9b0a0a6357aa2b0425a11c854ad8b40d9d8b444ab87abcd6aeaa82e"
Aug 04 03:21:43.788522 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:21:43.788502    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"642931b6a9b0a0a6357aa2b0425a11c854ad8b40d9d8b444ab87abcd6aeaa82e\": container with ID starting with 642931b6a9b0a0a6357aa2b0425a11c854ad8b40d9d8b444ab87abcd6aeaa82e not found: ID does not exist" containerID="642931b6a9b0a0a6357aa2b0425a11c854ad8b40d9d8b444ab87abcd6aeaa82e"
Aug 04 03:21:43.788585 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:43.788530    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642931b6a9b0a0a6357aa2b0425a11c854ad8b40d9d8b444ab87abcd6aeaa82e"} err="failed to get container status \"642931b6a9b0a0a6357aa2b0425a11c854ad8b40d9d8b444ab87abcd6aeaa82e\": rpc error: code = NotFound desc = could not find container \"642931b6a9b0a0a6357aa2b0425a11c854ad8b40d9d8b444ab87abcd6aeaa82e\": container with ID starting with 642931b6a9b0a0a6357aa2b0425a11c854ad8b40d9d8b444ab87abcd6aeaa82e not found: ID does not exist"
Aug 04 03:21:43.799711 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:43.799687    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d6db9d6fd-c4vlg"]
Aug 04 03:21:43.805048 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:43.805028    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d6db9d6fd-c4vlg"]
Aug 04 03:21:43.846664 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:21:43.846633    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e334ec4d-8a08-4d74-9a70-5b572b659b78" path="/var/lib/kubelet/pods/e334ec4d-8a08-4d74-9a70-5b572b659b78/volumes"
Aug 04 03:22:04.429985 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.429939    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7cdb695bfd-7xmds"]
Aug 04 03:22:04.430417 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.430303    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e334ec4d-8a08-4d74-9a70-5b572b659b78" containerName="manager"
Aug 04 03:22:04.430417 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.430316    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e334ec4d-8a08-4d74-9a70-5b572b659b78" containerName="manager"
Aug 04 03:22:04.430417 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.430405    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e334ec4d-8a08-4d74-9a70-5b572b659b78" containerName="manager"
Aug 04 03:22:04.438362 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.438343    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cdb695bfd-7xmds"
Aug 04 03:22:04.441658 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.441628    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Aug 04 03:22:04.441658 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.441631    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7z5pp\""
Aug 04 03:22:04.444374 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.444347    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cdb695bfd-7xmds"]
Aug 04 03:22:04.580396 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.580357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzcdz\" (UniqueName: \"kubernetes.io/projected/2e350ced-1eed-48f7-b2c3-bce2223db31a-kube-api-access-wzcdz\") pod \"maas-api-7cdb695bfd-7xmds\" (UID: \"2e350ced-1eed-48f7-b2c3-bce2223db31a\") " pod="odh-ai-gateway-infra/maas-api-7cdb695bfd-7xmds"
Aug 04 03:22:04.580571 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.580416    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e350ced-1eed-48f7-b2c3-bce2223db31a-openshift-service-ca\") pod \"maas-api-7cdb695bfd-7xmds\" (UID: \"2e350ced-1eed-48f7-b2c3-bce2223db31a\") " pod="odh-ai-gateway-infra/maas-api-7cdb695bfd-7xmds"
Aug 04 03:22:04.580571 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.580441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e350ced-1eed-48f7-b2c3-bce2223db31a-maas-api-tls\") pod \"maas-api-7cdb695bfd-7xmds\" (UID: \"2e350ced-1eed-48f7-b2c3-bce2223db31a\") " pod="odh-ai-gateway-infra/maas-api-7cdb695bfd-7xmds"
Aug 04 03:22:04.681932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.681843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzcdz\" (UniqueName: \"kubernetes.io/projected/2e350ced-1eed-48f7-b2c3-bce2223db31a-kube-api-access-wzcdz\") pod \"maas-api-7cdb695bfd-7xmds\" (UID: \"2e350ced-1eed-48f7-b2c3-bce2223db31a\") " pod="odh-ai-gateway-infra/maas-api-7cdb695bfd-7xmds"
Aug 04 03:22:04.681932 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.681906    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e350ced-1eed-48f7-b2c3-bce2223db31a-openshift-service-ca\") pod \"maas-api-7cdb695bfd-7xmds\" (UID: \"2e350ced-1eed-48f7-b2c3-bce2223db31a\") " pod="odh-ai-gateway-infra/maas-api-7cdb695bfd-7xmds"
Aug 04 03:22:04.682196 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.681936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e350ced-1eed-48f7-b2c3-bce2223db31a-maas-api-tls\") pod \"maas-api-7cdb695bfd-7xmds\" (UID: \"2e350ced-1eed-48f7-b2c3-bce2223db31a\") " pod="odh-ai-gateway-infra/maas-api-7cdb695bfd-7xmds"
Aug 04 03:22:04.682580 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.682558    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e350ced-1eed-48f7-b2c3-bce2223db31a-openshift-service-ca\") pod \"maas-api-7cdb695bfd-7xmds\" (UID: \"2e350ced-1eed-48f7-b2c3-bce2223db31a\") " pod="odh-ai-gateway-infra/maas-api-7cdb695bfd-7xmds"
Aug 04 03:22:04.684397 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.684376    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e350ced-1eed-48f7-b2c3-bce2223db31a-maas-api-tls\") pod \"maas-api-7cdb695bfd-7xmds\" (UID: \"2e350ced-1eed-48f7-b2c3-bce2223db31a\") " pod="odh-ai-gateway-infra/maas-api-7cdb695bfd-7xmds"
Aug 04 03:22:04.689729 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.689708    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzcdz\" (UniqueName: \"kubernetes.io/projected/2e350ced-1eed-48f7-b2c3-bce2223db31a-kube-api-access-wzcdz\") pod \"maas-api-7cdb695bfd-7xmds\" (UID: \"2e350ced-1eed-48f7-b2c3-bce2223db31a\") " pod="odh-ai-gateway-infra/maas-api-7cdb695bfd-7xmds"
Aug 04 03:22:04.751128 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.751092    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cdb695bfd-7xmds"
Aug 04 03:22:04.888591 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.888553    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cdb695bfd-7xmds"]
Aug 04 03:22:04.890732 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:22:04.890696    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e350ced_1eed_48f7_b2c3_bce2223db31a.slice/crio-28e0db036a740bb69197b3ec37c92567733c76d17a73472e87636c9511261857 WatchSource:0}: Error finding container 28e0db036a740bb69197b3ec37c92567733c76d17a73472e87636c9511261857: Status 404 returned error can't find the container with id 28e0db036a740bb69197b3ec37c92567733c76d17a73472e87636c9511261857
Aug 04 03:22:04.948348 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.948270    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-46wnj"]
Aug 04 03:22:04.953172 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.953156    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-46wnj"
Aug 04 03:22:04.955484 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.955464    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bmdrm\""
Aug 04 03:22:04.955586 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.955506    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Aug 04 03:22:04.968595 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:04.964408    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-46wnj"]
Aug 04 03:22:05.085276 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:05.085240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkzt5\" (UniqueName: \"kubernetes.io/projected/2dbe90ed-02ed-428f-ad98-ae60193a9e66-kube-api-access-rkzt5\") pod \"payload-processing-5cc6cd444b-46wnj\" (UID: \"2dbe90ed-02ed-428f-ad98-ae60193a9e66\") " pod="openshift-ingress/payload-processing-5cc6cd444b-46wnj"
Aug 04 03:22:05.085444 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:05.085321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbe90ed-02ed-428f-ad98-ae60193a9e66-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-46wnj\" (UID: \"2dbe90ed-02ed-428f-ad98-ae60193a9e66\") " pod="openshift-ingress/payload-processing-5cc6cd444b-46wnj"
Aug 04 03:22:05.185862 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:05.185825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbe90ed-02ed-428f-ad98-ae60193a9e66-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-46wnj\" (UID: \"2dbe90ed-02ed-428f-ad98-ae60193a9e66\") " pod="openshift-ingress/payload-processing-5cc6cd444b-46wnj"
Aug 04 03:22:05.186086 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:05.185874    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkzt5\" (UniqueName: \"kubernetes.io/projected/2dbe90ed-02ed-428f-ad98-ae60193a9e66-kube-api-access-rkzt5\") pod \"payload-processing-5cc6cd444b-46wnj\" (UID: \"2dbe90ed-02ed-428f-ad98-ae60193a9e66\") " pod="openshift-ingress/payload-processing-5cc6cd444b-46wnj"
Aug 04 03:22:05.186609 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:05.186582    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbe90ed-02ed-428f-ad98-ae60193a9e66-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-46wnj\" (UID: \"2dbe90ed-02ed-428f-ad98-ae60193a9e66\") " pod="openshift-ingress/payload-processing-5cc6cd444b-46wnj"
Aug 04 03:22:05.194167 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:05.194148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkzt5\" (UniqueName: \"kubernetes.io/projected/2dbe90ed-02ed-428f-ad98-ae60193a9e66-kube-api-access-rkzt5\") pod \"payload-processing-5cc6cd444b-46wnj\" (UID: \"2dbe90ed-02ed-428f-ad98-ae60193a9e66\") " pod="openshift-ingress/payload-processing-5cc6cd444b-46wnj"
Aug 04 03:22:05.268793 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:05.268757    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-46wnj"
Aug 04 03:22:05.400184 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:05.400149    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-46wnj"]
Aug 04 03:22:05.403243 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:22:05.403219    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbe90ed_02ed_428f_ad98_ae60193a9e66.slice/crio-dd674a08cbd517bce9584ef870593627f3a5c7e57d9246caf820ea3265bdb039 WatchSource:0}: Error finding container dd674a08cbd517bce9584ef870593627f3a5c7e57d9246caf820ea3265bdb039: Status 404 returned error can't find the container with id dd674a08cbd517bce9584ef870593627f3a5c7e57d9246caf820ea3265bdb039
Aug 04 03:22:05.884894 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:05.884852    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-46wnj" event={"ID":"2dbe90ed-02ed-428f-ad98-ae60193a9e66","Type":"ContainerStarted","Data":"dd674a08cbd517bce9584ef870593627f3a5c7e57d9246caf820ea3265bdb039"}
Aug 04 03:22:05.886250 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:05.886223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cdb695bfd-7xmds" event={"ID":"2e350ced-1eed-48f7-b2c3-bce2223db31a","Type":"ContainerStarted","Data":"28e0db036a740bb69197b3ec37c92567733c76d17a73472e87636c9511261857"}
Aug 04 03:22:06.892845 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:06.892809    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cdb695bfd-7xmds" event={"ID":"2e350ced-1eed-48f7-b2c3-bce2223db31a","Type":"ContainerStarted","Data":"5cc1dfbfaa7b083b0485cf555f30e926be9b52a0683d8b838d2377e7c020584e"}
Aug 04 03:22:06.893419 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:06.892948    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7cdb695bfd-7xmds"
Aug 04 03:22:06.912722 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:06.912669    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7cdb695bfd-7xmds" podStartSLOduration=1.391503551 podStartE2EDuration="2.912652433s" totalImagesPullingTime="1.521148882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:22:06.911447527 +0000 UTC m=+757.591234647" watchObservedRunningTime="2026-08-04 03:22:06.912652433 +0000 UTC m=+757.592439553"
Aug 04 03:22:07.898445 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:07.898409    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-46wnj" event={"ID":"2dbe90ed-02ed-428f-ad98-ae60193a9e66","Type":"ContainerStarted","Data":"5becbb7292374221ba93bb73147b3890e01173a3ee151f272830e9a72e4ad3b1"}
Aug 04 03:22:07.898878 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:07.898591    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-46wnj"
Aug 04 03:22:07.915280 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:07.915229    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-46wnj" podStartSLOduration=2.14672685 podStartE2EDuration="3.915211132s" totalImagesPullingTime="1.768484282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:22:07.913287438 +0000 UTC m=+758.593074575" watchObservedRunningTime="2026-08-04 03:22:07.915211132 +0000 UTC m=+758.594998251"
Aug 04 03:22:12.903906 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:12.903874    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7cdb695bfd-7xmds"
Aug 04 03:22:18.905152 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:18.905118    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-46wnj"
Aug 04 03:22:56.888736 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:56.888697    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"]
Aug 04 03:22:56.891188 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:56.891169    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:56.893528 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:56.893505    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Aug 04 03:22:56.893644 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:56.893588    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Aug 04 03:22:56.893644 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:56.893617    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jmw7d\""
Aug 04 03:22:56.893755 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:56.893620    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Aug 04 03:22:56.900150 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:56.900128    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"]
Aug 04 03:22:57.062282 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.062249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74775343-a44c-4b57-8430-d25999be311d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gvhmn\" (UID: \"74775343-a44c-4b57-8430-d25999be311d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:57.062486 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.062299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74775343-a44c-4b57-8430-d25999be311d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gvhmn\" (UID: \"74775343-a44c-4b57-8430-d25999be311d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:57.062486 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.062321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74775343-a44c-4b57-8430-d25999be311d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gvhmn\" (UID: \"74775343-a44c-4b57-8430-d25999be311d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:57.062486 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.062405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74775343-a44c-4b57-8430-d25999be311d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gvhmn\" (UID: \"74775343-a44c-4b57-8430-d25999be311d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:57.062486 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.062474    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74775343-a44c-4b57-8430-d25999be311d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gvhmn\" (UID: \"74775343-a44c-4b57-8430-d25999be311d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:57.062666 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.062571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kncws\" (UniqueName: \"kubernetes.io/projected/74775343-a44c-4b57-8430-d25999be311d-kube-api-access-kncws\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gvhmn\" (UID: \"74775343-a44c-4b57-8430-d25999be311d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:57.163563 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.163472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74775343-a44c-4b57-8430-d25999be311d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gvhmn\" (UID: \"74775343-a44c-4b57-8430-d25999be311d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:57.163563 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.163510    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74775343-a44c-4b57-8430-d25999be311d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gvhmn\" (UID: \"74775343-a44c-4b57-8430-d25999be311d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:57.163563 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.163544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74775343-a44c-4b57-8430-d25999be311d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gvhmn\" (UID: \"74775343-a44c-4b57-8430-d25999be311d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:57.163860 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.163597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74775343-a44c-4b57-8430-d25999be311d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gvhmn\" (UID: \"74775343-a44c-4b57-8430-d25999be311d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:57.163860 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.163659    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kncws\" (UniqueName: \"kubernetes.io/projected/74775343-a44c-4b57-8430-d25999be311d-kube-api-access-kncws\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gvhmn\" (UID: \"74775343-a44c-4b57-8430-d25999be311d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:57.163860 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.163692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74775343-a44c-4b57-8430-d25999be311d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gvhmn\" (UID: \"74775343-a44c-4b57-8430-d25999be311d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:57.164086 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.163937    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74775343-a44c-4b57-8430-d25999be311d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gvhmn\" (UID: \"74775343-a44c-4b57-8430-d25999be311d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:57.164086 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.164004    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74775343-a44c-4b57-8430-d25999be311d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gvhmn\" (UID: \"74775343-a44c-4b57-8430-d25999be311d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:57.164086 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.164032    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74775343-a44c-4b57-8430-d25999be311d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gvhmn\" (UID: \"74775343-a44c-4b57-8430-d25999be311d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:57.165836 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.165809    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74775343-a44c-4b57-8430-d25999be311d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gvhmn\" (UID: \"74775343-a44c-4b57-8430-d25999be311d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:57.166102 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.166085    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74775343-a44c-4b57-8430-d25999be311d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gvhmn\" (UID: \"74775343-a44c-4b57-8430-d25999be311d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:57.170887 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.170867    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kncws\" (UniqueName: \"kubernetes.io/projected/74775343-a44c-4b57-8430-d25999be311d-kube-api-access-kncws\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gvhmn\" (UID: \"74775343-a44c-4b57-8430-d25999be311d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:57.202773 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.202738    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:22:57.333823 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:57.333799    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"]
Aug 04 03:22:57.336137 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:22:57.336111    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74775343_a44c_4b57_8430_d25999be311d.slice/crio-06a95b4ae3fcc1bd3e8055756b2f136862ad6585b8e966c7015eb86570674949 WatchSource:0}: Error finding container 06a95b4ae3fcc1bd3e8055756b2f136862ad6585b8e966c7015eb86570674949: Status 404 returned error can't find the container with id 06a95b4ae3fcc1bd3e8055756b2f136862ad6585b8e966c7015eb86570674949
Aug 04 03:22:58.138323 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:58.138279    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn" event={"ID":"74775343-a44c-4b57-8430-d25999be311d","Type":"ContainerStarted","Data":"06a95b4ae3fcc1bd3e8055756b2f136862ad6585b8e966c7015eb86570674949"}
Aug 04 03:22:59.148919 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:59.148885    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn" event={"ID":"74775343-a44c-4b57-8430-d25999be311d","Type":"ContainerStarted","Data":"b8efde172a76011d269df20c38016a52281f033cb163a35ff6b353ea7e621d85"}
Aug 04 03:22:59.166122 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:22:59.166075    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn" podStartSLOduration=2.111289757 podStartE2EDuration="3.166060408s" totalImagesPullingTime="1.054770651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:22:59.164537908 +0000 UTC m=+809.844325052" watchObservedRunningTime="2026-08-04 03:22:59.166060408 +0000 UTC m=+809.845847580"
Aug 04 03:23:07.203539 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:07.203501    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:23:07.204034 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:07.203565    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:23:07.216209 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:07.216182    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:23:08.200007 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:08.199982    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gvhmn"
Aug 04 03:23:09.078700 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.078669    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"]
Aug 04 03:23:09.081429 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.081406    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.083887 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.083865    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Aug 04 03:23:09.089472 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.089453    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"]
Aug 04 03:23:09.178158 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.178123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4a7161-856a-4ca1-8fbb-7f5ebb94b266-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vchxc\" (UID: \"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.178158 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.178159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4a7161-856a-4ca1-8fbb-7f5ebb94b266-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vchxc\" (UID: \"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.178395 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.178184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4a7161-856a-4ca1-8fbb-7f5ebb94b266-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vchxc\" (UID: \"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.178395 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.178253    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4a7161-856a-4ca1-8fbb-7f5ebb94b266-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vchxc\" (UID: \"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.178395 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.178291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jz7x\" (UniqueName: \"kubernetes.io/projected/cd4a7161-856a-4ca1-8fbb-7f5ebb94b266-kube-api-access-8jz7x\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vchxc\" (UID: \"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.178538 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.178405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4a7161-856a-4ca1-8fbb-7f5ebb94b266-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vchxc\" (UID: \"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.279062 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.279024    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4a7161-856a-4ca1-8fbb-7f5ebb94b266-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vchxc\" (UID: \"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.279062 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.279070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4a7161-856a-4ca1-8fbb-7f5ebb94b266-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vchxc\" (UID: \"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.279297 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.279089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jz7x\" (UniqueName: \"kubernetes.io/projected/cd4a7161-856a-4ca1-8fbb-7f5ebb94b266-kube-api-access-8jz7x\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vchxc\" (UID: \"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.279297 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.279157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4a7161-856a-4ca1-8fbb-7f5ebb94b266-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vchxc\" (UID: \"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.279297 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.279253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4a7161-856a-4ca1-8fbb-7f5ebb94b266-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vchxc\" (UID: \"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.279297 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.279273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4a7161-856a-4ca1-8fbb-7f5ebb94b266-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vchxc\" (UID: \"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.279504 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.279487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4a7161-856a-4ca1-8fbb-7f5ebb94b266-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vchxc\" (UID: \"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.279566 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.279517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4a7161-856a-4ca1-8fbb-7f5ebb94b266-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vchxc\" (UID: \"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.279716 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.279693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4a7161-856a-4ca1-8fbb-7f5ebb94b266-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vchxc\" (UID: \"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.281548 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.281526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4a7161-856a-4ca1-8fbb-7f5ebb94b266-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vchxc\" (UID: \"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.281818 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.281799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4a7161-856a-4ca1-8fbb-7f5ebb94b266-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vchxc\" (UID: \"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.286921 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.286903    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jz7x\" (UniqueName: \"kubernetes.io/projected/cd4a7161-856a-4ca1-8fbb-7f5ebb94b266-kube-api-access-8jz7x\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vchxc\" (UID: \"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.393481 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.393391    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:09.526308 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:09.526284    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"]
Aug 04 03:23:09.528332 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:23:09.528300    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4a7161_856a_4ca1_8fbb_7f5ebb94b266.slice/crio-8cae08f2258326cd057e0b0b404bde9370fc168946b1bcde59d12c7dac72af41 WatchSource:0}: Error finding container 8cae08f2258326cd057e0b0b404bde9370fc168946b1bcde59d12c7dac72af41: Status 404 returned error can't find the container with id 8cae08f2258326cd057e0b0b404bde9370fc168946b1bcde59d12c7dac72af41
Aug 04 03:23:10.200715 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:10.200677    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc" event={"ID":"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266","Type":"ContainerStarted","Data":"c5edc2af24ccb8adf67ccc49525a157bbdcc8bb28322041fe8b5a9794689613a"}
Aug 04 03:23:10.201169 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:10.200729    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc" event={"ID":"cd4a7161-856a-4ca1-8fbb-7f5ebb94b266","Type":"ContainerStarted","Data":"8cae08f2258326cd057e0b0b404bde9370fc168946b1bcde59d12c7dac72af41"}
Aug 04 03:23:10.216258 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:10.216195    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc" podStartSLOduration=1.06272435 podStartE2EDuration="1.216174347s" totalImagesPullingTime="153.449997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:23:10.215474777 +0000 UTC m=+820.895261915" watchObservedRunningTime="2026-08-04 03:23:10.216174347 +0000 UTC m=+820.895961467"
Aug 04 03:23:19.394410 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:19.394368    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:19.394799 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:19.394436    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:19.407023 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:19.407000    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:23:20.260834 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:23:20.260807    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vchxc"
Aug 04 03:26:14.557066 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:26:14.557025    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5465dcb4d8-w2ksw"]
Aug 04 03:26:14.557592 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:26:14.557275    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5465dcb4d8-w2ksw" podUID="0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d" containerName="manager" containerID="cri-o://49b8b9785a644caa439eab78b33210ec1f8883156e8d703cb40d62d91e89aded" gracePeriod=30
Aug 04 03:26:14.811249 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:26:14.811188    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5465dcb4d8-w2ksw"
Aug 04 03:26:14.865582 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:26:14.865550    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpx6q\" (UniqueName: \"kubernetes.io/projected/0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d-kube-api-access-lpx6q\") pod \"0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d\" (UID: \"0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d\") "
Aug 04 03:26:14.865765 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:26:14.865598    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d-webhook-cert\") pod \"0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d\" (UID: \"0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d\") "
Aug 04 03:26:14.867567 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:26:14.867537    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d-kube-api-access-lpx6q" (OuterVolumeSpecName: "kube-api-access-lpx6q") pod "0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d" (UID: "0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d"). InnerVolumeSpecName "kube-api-access-lpx6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:26:14.867678 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:26:14.867596    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d" (UID: "0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:26:14.966802 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:26:14.966766    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpx6q\" (UniqueName: \"kubernetes.io/projected/0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d-kube-api-access-lpx6q\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:26:14.966802 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:26:14.966795    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d-webhook-cert\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:26:15.095177 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:26:15.095090    2579 generic.go:356] "Generic (PLEG): container finished" podID="0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d" containerID="49b8b9785a644caa439eab78b33210ec1f8883156e8d703cb40d62d91e89aded" exitCode=0
Aug 04 03:26:15.095177 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:26:15.095142    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5465dcb4d8-w2ksw"
Aug 04 03:26:15.095351 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:26:15.095181    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5465dcb4d8-w2ksw" event={"ID":"0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d","Type":"ContainerDied","Data":"49b8b9785a644caa439eab78b33210ec1f8883156e8d703cb40d62d91e89aded"}
Aug 04 03:26:15.095351 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:26:15.095237    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5465dcb4d8-w2ksw" event={"ID":"0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d","Type":"ContainerDied","Data":"e3ac2c1f5b4509d91bcbfc774f0df85389a94a85be52fde6fdbc32b2259a025f"}
Aug 04 03:26:15.095351 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:26:15.095261    2579 scope.go:117] "RemoveContainer" containerID="49b8b9785a644caa439eab78b33210ec1f8883156e8d703cb40d62d91e89aded"
Aug 04 03:26:15.104352 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:26:15.104338    2579 scope.go:117] "RemoveContainer" containerID="49b8b9785a644caa439eab78b33210ec1f8883156e8d703cb40d62d91e89aded"
Aug 04 03:26:15.104596 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:26:15.104579    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b8b9785a644caa439eab78b33210ec1f8883156e8d703cb40d62d91e89aded\": container with ID starting with 49b8b9785a644caa439eab78b33210ec1f8883156e8d703cb40d62d91e89aded not found: ID does not exist" containerID="49b8b9785a644caa439eab78b33210ec1f8883156e8d703cb40d62d91e89aded"
Aug 04 03:26:15.104635 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:26:15.104606    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b8b9785a644caa439eab78b33210ec1f8883156e8d703cb40d62d91e89aded"} err="failed to get container status \"49b8b9785a644caa439eab78b33210ec1f8883156e8d703cb40d62d91e89aded\": rpc error: code = NotFound desc = could not find container \"49b8b9785a644caa439eab78b33210ec1f8883156e8d703cb40d62d91e89aded\": container with ID starting with 49b8b9785a644caa439eab78b33210ec1f8883156e8d703cb40d62d91e89aded not found: ID does not exist"
Aug 04 03:26:15.115906 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:26:15.115844    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5465dcb4d8-w2ksw"]
Aug 04 03:26:15.117602 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:26:15.117582    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5465dcb4d8-w2ksw"]
Aug 04 03:26:15.845073 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:26:15.845043    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d" path="/var/lib/kubelet/pods/0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d/volumes"
Aug 04 03:27:15.880548 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:15.880516    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5465dcb4d8-b9k75"]
Aug 04 03:27:15.881031 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:15.880868    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d" containerName="manager"
Aug 04 03:27:15.881031 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:15.880878    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d" containerName="manager"
Aug 04 03:27:15.881031 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:15.880980    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f4530c0-c12b-4a6e-b76d-f5fa98b4a17d" containerName="manager"
Aug 04 03:27:15.884036 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:15.884022    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5465dcb4d8-b9k75"
Aug 04 03:27:15.887582 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:15.887560    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Aug 04 03:27:15.887708 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:15.887562    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Aug 04 03:27:15.887708 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:15.887561    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7jvk5\""
Aug 04 03:27:15.892040 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:15.892013    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5465dcb4d8-b9k75"]
Aug 04 03:27:16.030197 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:16.030156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk5j4\" (UniqueName: \"kubernetes.io/projected/18574208-bf6e-4025-97e8-befe8b70939c-kube-api-access-kk5j4\") pod \"maas-controller-5465dcb4d8-b9k75\" (UID: \"18574208-bf6e-4025-97e8-befe8b70939c\") " pod="opendatahub/maas-controller-5465dcb4d8-b9k75"
Aug 04 03:27:16.030387 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:16.030203    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18574208-bf6e-4025-97e8-befe8b70939c-webhook-cert\") pod \"maas-controller-5465dcb4d8-b9k75\" (UID: \"18574208-bf6e-4025-97e8-befe8b70939c\") " pod="opendatahub/maas-controller-5465dcb4d8-b9k75"
Aug 04 03:27:16.131807 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:16.131709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk5j4\" (UniqueName: \"kubernetes.io/projected/18574208-bf6e-4025-97e8-befe8b70939c-kube-api-access-kk5j4\") pod \"maas-controller-5465dcb4d8-b9k75\" (UID: \"18574208-bf6e-4025-97e8-befe8b70939c\") " pod="opendatahub/maas-controller-5465dcb4d8-b9k75"
Aug 04 03:27:16.131807 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:16.131761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18574208-bf6e-4025-97e8-befe8b70939c-webhook-cert\") pod \"maas-controller-5465dcb4d8-b9k75\" (UID: \"18574208-bf6e-4025-97e8-befe8b70939c\") " pod="opendatahub/maas-controller-5465dcb4d8-b9k75"
Aug 04 03:27:16.134224 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:16.134195    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18574208-bf6e-4025-97e8-befe8b70939c-webhook-cert\") pod \"maas-controller-5465dcb4d8-b9k75\" (UID: \"18574208-bf6e-4025-97e8-befe8b70939c\") " pod="opendatahub/maas-controller-5465dcb4d8-b9k75"
Aug 04 03:27:16.143476 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:16.143451    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk5j4\" (UniqueName: \"kubernetes.io/projected/18574208-bf6e-4025-97e8-befe8b70939c-kube-api-access-kk5j4\") pod \"maas-controller-5465dcb4d8-b9k75\" (UID: \"18574208-bf6e-4025-97e8-befe8b70939c\") " pod="opendatahub/maas-controller-5465dcb4d8-b9k75"
Aug 04 03:27:16.196292 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:16.196262    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5465dcb4d8-b9k75"
Aug 04 03:27:16.327500 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:16.327473    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5465dcb4d8-b9k75"]
Aug 04 03:27:16.328910 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:27:16.328877    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18574208_bf6e_4025_97e8_befe8b70939c.slice/crio-fc808267db4bbd70fc6dc101bd4c2a3aa74a60f6b286c69946203151befeb2b2 WatchSource:0}: Error finding container fc808267db4bbd70fc6dc101bd4c2a3aa74a60f6b286c69946203151befeb2b2: Status 404 returned error can't find the container with id fc808267db4bbd70fc6dc101bd4c2a3aa74a60f6b286c69946203151befeb2b2
Aug 04 03:27:16.330283 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:16.330264    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 03:27:16.388135 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:16.388059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5465dcb4d8-b9k75" event={"ID":"18574208-bf6e-4025-97e8-befe8b70939c","Type":"ContainerStarted","Data":"fc808267db4bbd70fc6dc101bd4c2a3aa74a60f6b286c69946203151befeb2b2"}
Aug 04 03:27:17.396126 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:17.396042    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5465dcb4d8-b9k75" event={"ID":"18574208-bf6e-4025-97e8-befe8b70939c","Type":"ContainerStarted","Data":"d0798d514fdda15784536a83153951d4f1c51808dd63cfa38c49d8e7191f8f65"}
Aug 04 03:27:17.396479 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:17.396163    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5465dcb4d8-b9k75"
Aug 04 03:27:17.413318 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:17.413267    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5465dcb4d8-b9k75" podStartSLOduration=1.704023045 podStartE2EDuration="2.413251822s" totalImagesPullingTime="709.228777ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:27:17.41308578 +0000 UTC m=+1068.092872915" watchObservedRunningTime="2026-08-04 03:27:17.413251822 +0000 UTC m=+1068.093038940"
Aug 04 03:27:28.402841 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:28.402753    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5465dcb4d8-b9k75"
Aug 04 03:27:33.007649 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:33.007612    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5465dcb4d8-b9k75"]
Aug 04 03:27:33.008141 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:33.007823    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5465dcb4d8-b9k75" podUID="18574208-bf6e-4025-97e8-befe8b70939c" containerName="manager" containerID="cri-o://d0798d514fdda15784536a83153951d4f1c51808dd63cfa38c49d8e7191f8f65" gracePeriod=30
Aug 04 03:27:33.252892 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:33.252867    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5465dcb4d8-b9k75"
Aug 04 03:27:33.284995 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:33.284898    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18574208-bf6e-4025-97e8-befe8b70939c-webhook-cert\") pod \"18574208-bf6e-4025-97e8-befe8b70939c\" (UID: \"18574208-bf6e-4025-97e8-befe8b70939c\") "
Aug 04 03:27:33.285123 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:33.285037    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk5j4\" (UniqueName: \"kubernetes.io/projected/18574208-bf6e-4025-97e8-befe8b70939c-kube-api-access-kk5j4\") pod \"18574208-bf6e-4025-97e8-befe8b70939c\" (UID: \"18574208-bf6e-4025-97e8-befe8b70939c\") "
Aug 04 03:27:33.286911 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:33.286882    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18574208-bf6e-4025-97e8-befe8b70939c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "18574208-bf6e-4025-97e8-befe8b70939c" (UID: "18574208-bf6e-4025-97e8-befe8b70939c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:27:33.287116 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:33.287089    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18574208-bf6e-4025-97e8-befe8b70939c-kube-api-access-kk5j4" (OuterVolumeSpecName: "kube-api-access-kk5j4") pod "18574208-bf6e-4025-97e8-befe8b70939c" (UID: "18574208-bf6e-4025-97e8-befe8b70939c"). InnerVolumeSpecName "kube-api-access-kk5j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:27:33.386145 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:33.386110    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18574208-bf6e-4025-97e8-befe8b70939c-webhook-cert\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:27:33.386145 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:33.386138    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk5j4\" (UniqueName: \"kubernetes.io/projected/18574208-bf6e-4025-97e8-befe8b70939c-kube-api-access-kk5j4\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:27:33.473495 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:33.473464    2579 generic.go:356] "Generic (PLEG): container finished" podID="18574208-bf6e-4025-97e8-befe8b70939c" containerID="d0798d514fdda15784536a83153951d4f1c51808dd63cfa38c49d8e7191f8f65" exitCode=0
Aug 04 03:27:33.473660 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:33.473523    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5465dcb4d8-b9k75"
Aug 04 03:27:33.473660 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:33.473553    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5465dcb4d8-b9k75" event={"ID":"18574208-bf6e-4025-97e8-befe8b70939c","Type":"ContainerDied","Data":"d0798d514fdda15784536a83153951d4f1c51808dd63cfa38c49d8e7191f8f65"}
Aug 04 03:27:33.473660 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:33.473614    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5465dcb4d8-b9k75" event={"ID":"18574208-bf6e-4025-97e8-befe8b70939c","Type":"ContainerDied","Data":"fc808267db4bbd70fc6dc101bd4c2a3aa74a60f6b286c69946203151befeb2b2"}
Aug 04 03:27:33.473660 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:33.473644    2579 scope.go:117] "RemoveContainer" containerID="d0798d514fdda15784536a83153951d4f1c51808dd63cfa38c49d8e7191f8f65"
Aug 04 03:27:33.483016 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:33.482999    2579 scope.go:117] "RemoveContainer" containerID="d0798d514fdda15784536a83153951d4f1c51808dd63cfa38c49d8e7191f8f65"
Aug 04 03:27:33.483274 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:27:33.483247    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0798d514fdda15784536a83153951d4f1c51808dd63cfa38c49d8e7191f8f65\": container with ID starting with d0798d514fdda15784536a83153951d4f1c51808dd63cfa38c49d8e7191f8f65 not found: ID does not exist" containerID="d0798d514fdda15784536a83153951d4f1c51808dd63cfa38c49d8e7191f8f65"
Aug 04 03:27:33.483325 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:33.483286    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0798d514fdda15784536a83153951d4f1c51808dd63cfa38c49d8e7191f8f65"} err="failed to get container status \"d0798d514fdda15784536a83153951d4f1c51808dd63cfa38c49d8e7191f8f65\": rpc error: code = NotFound desc = could not find container \"d0798d514fdda15784536a83153951d4f1c51808dd63cfa38c49d8e7191f8f65\": container with ID starting with d0798d514fdda15784536a83153951d4f1c51808dd63cfa38c49d8e7191f8f65 not found: ID does not exist"
Aug 04 03:27:33.495027 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:33.495007    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5465dcb4d8-b9k75"]
Aug 04 03:27:33.499213 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:33.499193    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5465dcb4d8-b9k75"]
Aug 04 03:27:33.845219 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:33.845188    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18574208-bf6e-4025-97e8-befe8b70939c" path="/var/lib/kubelet/pods/18574208-bf6e-4025-97e8-befe8b70939c/volumes"
Aug 04 03:27:34.251830 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:34.251788    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5465dcb4d8-z7qrk"]
Aug 04 03:27:34.252381 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:34.252362    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18574208-bf6e-4025-97e8-befe8b70939c" containerName="manager"
Aug 04 03:27:34.252461 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:34.252385    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="18574208-bf6e-4025-97e8-befe8b70939c" containerName="manager"
Aug 04 03:27:34.252538 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:34.252525    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="18574208-bf6e-4025-97e8-befe8b70939c" containerName="manager"
Aug 04 03:27:34.256794 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:34.256772    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5465dcb4d8-z7qrk"
Aug 04 03:27:34.260015 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:34.259996    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Aug 04 03:27:34.260250 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:34.259996    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7jvk5\""
Aug 04 03:27:34.260250 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:34.260033    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Aug 04 03:27:34.263079 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:34.263050    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5465dcb4d8-z7qrk"]
Aug 04 03:27:34.293451 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:34.293419    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a292744-1316-4511-bd59-4a1b67a2d714-webhook-cert\") pod \"maas-controller-5465dcb4d8-z7qrk\" (UID: \"9a292744-1316-4511-bd59-4a1b67a2d714\") " pod="opendatahub/maas-controller-5465dcb4d8-z7qrk"
Aug 04 03:27:34.293607 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:34.293530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khdfq\" (UniqueName: \"kubernetes.io/projected/9a292744-1316-4511-bd59-4a1b67a2d714-kube-api-access-khdfq\") pod \"maas-controller-5465dcb4d8-z7qrk\" (UID: \"9a292744-1316-4511-bd59-4a1b67a2d714\") " pod="opendatahub/maas-controller-5465dcb4d8-z7qrk"
Aug 04 03:27:34.394295 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:34.394260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khdfq\" (UniqueName: \"kubernetes.io/projected/9a292744-1316-4511-bd59-4a1b67a2d714-kube-api-access-khdfq\") pod \"maas-controller-5465dcb4d8-z7qrk\" (UID: \"9a292744-1316-4511-bd59-4a1b67a2d714\") " pod="opendatahub/maas-controller-5465dcb4d8-z7qrk"
Aug 04 03:27:34.394493 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:34.394317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a292744-1316-4511-bd59-4a1b67a2d714-webhook-cert\") pod \"maas-controller-5465dcb4d8-z7qrk\" (UID: \"9a292744-1316-4511-bd59-4a1b67a2d714\") " pod="opendatahub/maas-controller-5465dcb4d8-z7qrk"
Aug 04 03:27:34.396749 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:34.396724    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a292744-1316-4511-bd59-4a1b67a2d714-webhook-cert\") pod \"maas-controller-5465dcb4d8-z7qrk\" (UID: \"9a292744-1316-4511-bd59-4a1b67a2d714\") " pod="opendatahub/maas-controller-5465dcb4d8-z7qrk"
Aug 04 03:27:34.401709 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:34.401685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khdfq\" (UniqueName: \"kubernetes.io/projected/9a292744-1316-4511-bd59-4a1b67a2d714-kube-api-access-khdfq\") pod \"maas-controller-5465dcb4d8-z7qrk\" (UID: \"9a292744-1316-4511-bd59-4a1b67a2d714\") " pod="opendatahub/maas-controller-5465dcb4d8-z7qrk"
Aug 04 03:27:34.567888 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:34.567804    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5465dcb4d8-z7qrk"
Aug 04 03:27:34.696288 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:34.696255    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5465dcb4d8-z7qrk"]
Aug 04 03:27:34.699333 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:27:34.699306    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a292744_1316_4511_bd59_4a1b67a2d714.slice/crio-9c6d9bcdc85a0655f2cb7c1cf1c4de24daac891e21ab6588ca934e0a3a3b1dfb WatchSource:0}: Error finding container 9c6d9bcdc85a0655f2cb7c1cf1c4de24daac891e21ab6588ca934e0a3a3b1dfb: Status 404 returned error can't find the container with id 9c6d9bcdc85a0655f2cb7c1cf1c4de24daac891e21ab6588ca934e0a3a3b1dfb
Aug 04 03:27:35.484494 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:35.484463    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5465dcb4d8-z7qrk" event={"ID":"9a292744-1316-4511-bd59-4a1b67a2d714","Type":"ContainerStarted","Data":"e831df6417f2642e681bddf0e2d42a17d8253374c1f85b41bb689e999e3c8999"}
Aug 04 03:27:35.484875 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:35.484517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5465dcb4d8-z7qrk" event={"ID":"9a292744-1316-4511-bd59-4a1b67a2d714","Type":"ContainerStarted","Data":"9c6d9bcdc85a0655f2cb7c1cf1c4de24daac891e21ab6588ca934e0a3a3b1dfb"}
Aug 04 03:27:35.484875 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:35.484538    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5465dcb4d8-z7qrk"
Aug 04 03:27:35.502792 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:35.502745    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5465dcb4d8-z7qrk" podStartSLOduration=1.199058432 podStartE2EDuration="1.502727497s" totalImagesPullingTime="303.669065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:27:35.501072931 +0000 UTC m=+1086.180860066" watchObservedRunningTime="2026-08-04 03:27:35.502727497 +0000 UTC m=+1086.182514748"
Aug 04 03:27:46.491315 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:27:46.491284    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5465dcb4d8-z7qrk"
Aug 04 03:44:48.900912 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:48.900880    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"]
Aug 04 03:44:48.904597 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:48.904575    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"
Aug 04 03:44:48.907538 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:48.907514    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-49bbb41b-serving-cert\""
Aug 04 03:44:48.921363 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:48.921336    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"]
Aug 04 03:44:49.016415 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.016371    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwhg6\" (UniqueName: \"kubernetes.io/projected/f390fc39-1de3-454e-84d1-f0eba4ecda25-kube-api-access-pwhg6\") pod \"maas-api-e2e-ait-49bbb41b-6775cd99-mm22b\" (UID: \"f390fc39-1de3-454e-84d1-f0eba4ecda25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"
Aug 04 03:44:49.016598 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.016422    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f390fc39-1de3-454e-84d1-f0eba4ecda25-maas-api-tls\") pod \"maas-api-e2e-ait-49bbb41b-6775cd99-mm22b\" (UID: \"f390fc39-1de3-454e-84d1-f0eba4ecda25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"
Aug 04 03:44:49.016598 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.016469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f390fc39-1de3-454e-84d1-f0eba4ecda25-openshift-service-ca\") pod \"maas-api-e2e-ait-49bbb41b-6775cd99-mm22b\" (UID: \"f390fc39-1de3-454e-84d1-f0eba4ecda25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"
Aug 04 03:44:49.117809 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.117763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwhg6\" (UniqueName: \"kubernetes.io/projected/f390fc39-1de3-454e-84d1-f0eba4ecda25-kube-api-access-pwhg6\") pod \"maas-api-e2e-ait-49bbb41b-6775cd99-mm22b\" (UID: \"f390fc39-1de3-454e-84d1-f0eba4ecda25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"
Aug 04 03:44:49.117809 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.117813    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f390fc39-1de3-454e-84d1-f0eba4ecda25-maas-api-tls\") pod \"maas-api-e2e-ait-49bbb41b-6775cd99-mm22b\" (UID: \"f390fc39-1de3-454e-84d1-f0eba4ecda25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"
Aug 04 03:44:49.118043 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.117834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f390fc39-1de3-454e-84d1-f0eba4ecda25-openshift-service-ca\") pod \"maas-api-e2e-ait-49bbb41b-6775cd99-mm22b\" (UID: \"f390fc39-1de3-454e-84d1-f0eba4ecda25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"
Aug 04 03:44:49.118043 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:44:49.117938    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-serving-cert: secret "maas-api-e2e-ait-49bbb41b-serving-cert" not found
Aug 04 03:44:49.118043 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:44:49.118031    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f390fc39-1de3-454e-84d1-f0eba4ecda25-maas-api-tls podName:f390fc39-1de3-454e-84d1-f0eba4ecda25 nodeName:}" failed. No retries permitted until 2026-08-04 03:44:49.618006741 +0000 UTC m=+2120.297793847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f390fc39-1de3-454e-84d1-f0eba4ecda25-maas-api-tls") pod "maas-api-e2e-ait-49bbb41b-6775cd99-mm22b" (UID: "f390fc39-1de3-454e-84d1-f0eba4ecda25") : secret "maas-api-e2e-ait-49bbb41b-serving-cert" not found
Aug 04 03:44:49.118564 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.118546    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f390fc39-1de3-454e-84d1-f0eba4ecda25-openshift-service-ca\") pod \"maas-api-e2e-ait-49bbb41b-6775cd99-mm22b\" (UID: \"f390fc39-1de3-454e-84d1-f0eba4ecda25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"
Aug 04 03:44:49.126718 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.126692    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwhg6\" (UniqueName: \"kubernetes.io/projected/f390fc39-1de3-454e-84d1-f0eba4ecda25-kube-api-access-pwhg6\") pod \"maas-api-e2e-ait-49bbb41b-6775cd99-mm22b\" (UID: \"f390fc39-1de3-454e-84d1-f0eba4ecda25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"
Aug 04 03:44:49.507589 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.507553    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d"]
Aug 04 03:44:49.511582 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.511563    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d"
Aug 04 03:44:49.514112 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.514089    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-49bbb41b\""
Aug 04 03:44:49.518079 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.518045    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d"]
Aug 04 03:44:49.624268 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.624232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be9f52b-ccab-4249-890c-be03403ee233-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d\" (UID: \"5be9f52b-ccab-4249-890c-be03403ee233\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d"
Aug 04 03:44:49.624434 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.624320    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvzt2\" (UniqueName: \"kubernetes.io/projected/5be9f52b-ccab-4249-890c-be03403ee233-kube-api-access-qvzt2\") pod \"payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d\" (UID: \"5be9f52b-ccab-4249-890c-be03403ee233\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d"
Aug 04 03:44:49.624434 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.624347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f390fc39-1de3-454e-84d1-f0eba4ecda25-maas-api-tls\") pod \"maas-api-e2e-ait-49bbb41b-6775cd99-mm22b\" (UID: \"f390fc39-1de3-454e-84d1-f0eba4ecda25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"
Aug 04 03:44:49.626897 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.626876    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f390fc39-1de3-454e-84d1-f0eba4ecda25-maas-api-tls\") pod \"maas-api-e2e-ait-49bbb41b-6775cd99-mm22b\" (UID: \"f390fc39-1de3-454e-84d1-f0eba4ecda25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"
Aug 04 03:44:49.725519 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.725475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be9f52b-ccab-4249-890c-be03403ee233-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d\" (UID: \"5be9f52b-ccab-4249-890c-be03403ee233\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d"
Aug 04 03:44:49.725725 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.725574    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvzt2\" (UniqueName: \"kubernetes.io/projected/5be9f52b-ccab-4249-890c-be03403ee233-kube-api-access-qvzt2\") pod \"payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d\" (UID: \"5be9f52b-ccab-4249-890c-be03403ee233\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d"
Aug 04 03:44:49.726236 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.726217    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be9f52b-ccab-4249-890c-be03403ee233-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d\" (UID: \"5be9f52b-ccab-4249-890c-be03403ee233\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d"
Aug 04 03:44:49.733131 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.733109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvzt2\" (UniqueName: \"kubernetes.io/projected/5be9f52b-ccab-4249-890c-be03403ee233-kube-api-access-qvzt2\") pod \"payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d\" (UID: \"5be9f52b-ccab-4249-890c-be03403ee233\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d"
Aug 04 03:44:49.815555 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.815467    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"
Aug 04 03:44:49.823836 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.823806    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d"
Aug 04 03:44:49.969834 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.969802    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"]
Aug 04 03:44:49.972173 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:44:49.972136    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf390fc39_1de3_454e_84d1_f0eba4ecda25.slice/crio-8bc17897c96d281e834cb5c5c9fd1dd75e7871e66c6e39bad9a6ca68fb29b2bd WatchSource:0}: Error finding container 8bc17897c96d281e834cb5c5c9fd1dd75e7871e66c6e39bad9a6ca68fb29b2bd: Status 404 returned error can't find the container with id 8bc17897c96d281e834cb5c5c9fd1dd75e7871e66c6e39bad9a6ca68fb29b2bd
Aug 04 03:44:49.975800 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.974262    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 03:44:49.990982 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:49.990935    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d"]
Aug 04 03:44:49.992814 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:44:49.992789    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be9f52b_ccab_4249_890c_be03403ee233.slice/crio-90a7adb180699d22c9d3667643369e9dc3b63df32b70789135e0dde049bfc171 WatchSource:0}: Error finding container 90a7adb180699d22c9d3667643369e9dc3b63df32b70789135e0dde049bfc171: Status 404 returned error can't find the container with id 90a7adb180699d22c9d3667643369e9dc3b63df32b70789135e0dde049bfc171
Aug 04 03:44:50.456643 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:50.456605    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d" event={"ID":"5be9f52b-ccab-4249-890c-be03403ee233","Type":"ContainerStarted","Data":"5d9211f7f1c9b08419e418c8795661e63dff5b1a63f0eba9520db8cb15b8abf2"}
Aug 04 03:44:50.456858 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:50.456674    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d" event={"ID":"5be9f52b-ccab-4249-890c-be03403ee233","Type":"ContainerStarted","Data":"90a7adb180699d22c9d3667643369e9dc3b63df32b70789135e0dde049bfc171"}
Aug 04 03:44:50.456858 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:50.456803    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d"
Aug 04 03:44:50.458110 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:50.458086    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b" event={"ID":"f390fc39-1de3-454e-84d1-f0eba4ecda25","Type":"ContainerStarted","Data":"8bc17897c96d281e834cb5c5c9fd1dd75e7871e66c6e39bad9a6ca68fb29b2bd"}
Aug 04 03:44:50.477109 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:50.477061    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d" podStartSLOduration=1.477046625 podStartE2EDuration="1.477046625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:44:50.47566636 +0000 UTC m=+2121.155453504" watchObservedRunningTime="2026-08-04 03:44:50.477046625 +0000 UTC m=+2121.156833743"
Aug 04 03:44:51.465133 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:51.465096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b" event={"ID":"f390fc39-1de3-454e-84d1-f0eba4ecda25","Type":"ContainerStarted","Data":"fdef81fea34d8ceb7c6eaa75458c6a3cc1feb87893d0bc705b73f5a9cb68f35a"}
Aug 04 03:44:51.483003 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:51.482937    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b" podStartSLOduration=2.942443522 podStartE2EDuration="3.482921786s" totalImagesPullingTime="540.478264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:44:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:44:51.481051491 +0000 UTC m=+2122.160838641" watchObservedRunningTime="2026-08-04 03:44:51.482921786 +0000 UTC m=+2122.162708904"
Aug 04 03:44:52.469930 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:52.469898    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"
Aug 04 03:44:58.480106 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:44:58.480077    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"
Aug 04 03:45:01.467951 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:01.467915    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d"
Aug 04 03:45:19.239606 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.239567    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"]
Aug 04 03:45:19.240283 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.239858    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b" podUID="f390fc39-1de3-454e-84d1-f0eba4ecda25" containerName="maas-api" containerID="cri-o://fdef81fea34d8ceb7c6eaa75458c6a3cc1feb87893d0bc705b73f5a9cb68f35a" gracePeriod=30
Aug 04 03:45:19.300841 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.300804    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d"]
Aug 04 03:45:19.301166 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.301076    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d" podUID="5be9f52b-ccab-4249-890c-be03403ee233" containerName="payload-pre-processing" containerID="cri-o://5d9211f7f1c9b08419e418c8795661e63dff5b1a63f0eba9520db8cb15b8abf2" gracePeriod=30
Aug 04 03:45:19.571947 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.571921    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"
Aug 04 03:45:19.578361 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.578331    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d"
Aug 04 03:45:19.597374 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.597351    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f390fc39-1de3-454e-84d1-f0eba4ecda25-maas-api-tls\") pod \"f390fc39-1de3-454e-84d1-f0eba4ecda25\" (UID: \"f390fc39-1de3-454e-84d1-f0eba4ecda25\") "
Aug 04 03:45:19.597493 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.597383    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvzt2\" (UniqueName: \"kubernetes.io/projected/5be9f52b-ccab-4249-890c-be03403ee233-kube-api-access-qvzt2\") pod \"5be9f52b-ccab-4249-890c-be03403ee233\" (UID: \"5be9f52b-ccab-4249-890c-be03403ee233\") "
Aug 04 03:45:19.597493 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.597416    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f390fc39-1de3-454e-84d1-f0eba4ecda25-openshift-service-ca\") pod \"f390fc39-1de3-454e-84d1-f0eba4ecda25\" (UID: \"f390fc39-1de3-454e-84d1-f0eba4ecda25\") "
Aug 04 03:45:19.597576 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.597513    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwhg6\" (UniqueName: \"kubernetes.io/projected/f390fc39-1de3-454e-84d1-f0eba4ecda25-kube-api-access-pwhg6\") pod \"f390fc39-1de3-454e-84d1-f0eba4ecda25\" (UID: \"f390fc39-1de3-454e-84d1-f0eba4ecda25\") "
Aug 04 03:45:19.597576 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.597551    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be9f52b-ccab-4249-890c-be03403ee233-plugins-config-volume\") pod \"5be9f52b-ccab-4249-890c-be03403ee233\" (UID: \"5be9f52b-ccab-4249-890c-be03403ee233\") "
Aug 04 03:45:19.597881 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.597850    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f390fc39-1de3-454e-84d1-f0eba4ecda25-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f390fc39-1de3-454e-84d1-f0eba4ecda25" (UID: "f390fc39-1de3-454e-84d1-f0eba4ecda25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:45:19.598004 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.597952    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be9f52b-ccab-4249-890c-be03403ee233-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5be9f52b-ccab-4249-890c-be03403ee233" (UID: "5be9f52b-ccab-4249-890c-be03403ee233"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:45:19.599697 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.599667    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f390fc39-1de3-454e-84d1-f0eba4ecda25-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f390fc39-1de3-454e-84d1-f0eba4ecda25" (UID: "f390fc39-1de3-454e-84d1-f0eba4ecda25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:45:19.599780 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.599726    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f390fc39-1de3-454e-84d1-f0eba4ecda25-kube-api-access-pwhg6" (OuterVolumeSpecName: "kube-api-access-pwhg6") pod "f390fc39-1de3-454e-84d1-f0eba4ecda25" (UID: "f390fc39-1de3-454e-84d1-f0eba4ecda25"). InnerVolumeSpecName "kube-api-access-pwhg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:45:19.599780 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.599764    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be9f52b-ccab-4249-890c-be03403ee233-kube-api-access-qvzt2" (OuterVolumeSpecName: "kube-api-access-qvzt2") pod "5be9f52b-ccab-4249-890c-be03403ee233" (UID: "5be9f52b-ccab-4249-890c-be03403ee233"). InnerVolumeSpecName "kube-api-access-qvzt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:45:19.618601 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.618571    2579 generic.go:356] "Generic (PLEG): container finished" podID="5be9f52b-ccab-4249-890c-be03403ee233" containerID="5d9211f7f1c9b08419e418c8795661e63dff5b1a63f0eba9520db8cb15b8abf2" exitCode=0
Aug 04 03:45:19.618753 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.618643    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d"
Aug 04 03:45:19.618753 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.618665    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d" event={"ID":"5be9f52b-ccab-4249-890c-be03403ee233","Type":"ContainerDied","Data":"5d9211f7f1c9b08419e418c8795661e63dff5b1a63f0eba9520db8cb15b8abf2"}
Aug 04 03:45:19.618876 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.618759    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d" event={"ID":"5be9f52b-ccab-4249-890c-be03403ee233","Type":"ContainerDied","Data":"90a7adb180699d22c9d3667643369e9dc3b63df32b70789135e0dde049bfc171"}
Aug 04 03:45:19.618876 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.618794    2579 scope.go:117] "RemoveContainer" containerID="5d9211f7f1c9b08419e418c8795661e63dff5b1a63f0eba9520db8cb15b8abf2"
Aug 04 03:45:19.620351 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.620327    2579 generic.go:356] "Generic (PLEG): container finished" podID="f390fc39-1de3-454e-84d1-f0eba4ecda25" containerID="fdef81fea34d8ceb7c6eaa75458c6a3cc1feb87893d0bc705b73f5a9cb68f35a" exitCode=0
Aug 04 03:45:19.620462 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.620368    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b" event={"ID":"f390fc39-1de3-454e-84d1-f0eba4ecda25","Type":"ContainerDied","Data":"fdef81fea34d8ceb7c6eaa75458c6a3cc1feb87893d0bc705b73f5a9cb68f35a"}
Aug 04 03:45:19.620462 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.620400    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b" event={"ID":"f390fc39-1de3-454e-84d1-f0eba4ecda25","Type":"ContainerDied","Data":"8bc17897c96d281e834cb5c5c9fd1dd75e7871e66c6e39bad9a6ca68fb29b2bd"}
Aug 04 03:45:19.620578 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.620467    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"
Aug 04 03:45:19.629207 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.629185    2579 scope.go:117] "RemoveContainer" containerID="5d9211f7f1c9b08419e418c8795661e63dff5b1a63f0eba9520db8cb15b8abf2"
Aug 04 03:45:19.629554 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:45:19.629533    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9211f7f1c9b08419e418c8795661e63dff5b1a63f0eba9520db8cb15b8abf2\": container with ID starting with 5d9211f7f1c9b08419e418c8795661e63dff5b1a63f0eba9520db8cb15b8abf2 not found: ID does not exist" containerID="5d9211f7f1c9b08419e418c8795661e63dff5b1a63f0eba9520db8cb15b8abf2"
Aug 04 03:45:19.629634 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.629566    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9211f7f1c9b08419e418c8795661e63dff5b1a63f0eba9520db8cb15b8abf2"} err="failed to get container status \"5d9211f7f1c9b08419e418c8795661e63dff5b1a63f0eba9520db8cb15b8abf2\": rpc error: code = NotFound desc = could not find container \"5d9211f7f1c9b08419e418c8795661e63dff5b1a63f0eba9520db8cb15b8abf2\": container with ID starting with 5d9211f7f1c9b08419e418c8795661e63dff5b1a63f0eba9520db8cb15b8abf2 not found: ID does not exist"
Aug 04 03:45:19.629634 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.629590    2579 scope.go:117] "RemoveContainer" containerID="fdef81fea34d8ceb7c6eaa75458c6a3cc1feb87893d0bc705b73f5a9cb68f35a"
Aug 04 03:45:19.639247 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.639221    2579 scope.go:117] "RemoveContainer" containerID="fdef81fea34d8ceb7c6eaa75458c6a3cc1feb87893d0bc705b73f5a9cb68f35a"
Aug 04 03:45:19.639498 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:45:19.639478    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdef81fea34d8ceb7c6eaa75458c6a3cc1feb87893d0bc705b73f5a9cb68f35a\": container with ID starting with fdef81fea34d8ceb7c6eaa75458c6a3cc1feb87893d0bc705b73f5a9cb68f35a not found: ID does not exist" containerID="fdef81fea34d8ceb7c6eaa75458c6a3cc1feb87893d0bc705b73f5a9cb68f35a"
Aug 04 03:45:19.639551 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.639506    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdef81fea34d8ceb7c6eaa75458c6a3cc1feb87893d0bc705b73f5a9cb68f35a"} err="failed to get container status \"fdef81fea34d8ceb7c6eaa75458c6a3cc1feb87893d0bc705b73f5a9cb68f35a\": rpc error: code = NotFound desc = could not find container \"fdef81fea34d8ceb7c6eaa75458c6a3cc1feb87893d0bc705b73f5a9cb68f35a\": container with ID starting with fdef81fea34d8ceb7c6eaa75458c6a3cc1feb87893d0bc705b73f5a9cb68f35a not found: ID does not exist"
Aug 04 03:45:19.645240 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.645220    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d"]
Aug 04 03:45:19.649484 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.649465    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49bbb41b-788895f5b4-86s4d"]
Aug 04 03:45:19.660606 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.660586    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"]
Aug 04 03:45:19.664226 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.664207    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49bbb41b-6775cd99-mm22b"]
Aug 04 03:45:19.698656 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.698634    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwhg6\" (UniqueName: \"kubernetes.io/projected/f390fc39-1de3-454e-84d1-f0eba4ecda25-kube-api-access-pwhg6\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:45:19.698656 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.698657    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be9f52b-ccab-4249-890c-be03403ee233-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:45:19.698786 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.698669    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f390fc39-1de3-454e-84d1-f0eba4ecda25-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:45:19.698786 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.698678    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvzt2\" (UniqueName: \"kubernetes.io/projected/5be9f52b-ccab-4249-890c-be03403ee233-kube-api-access-qvzt2\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:45:19.698786 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.698686    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f390fc39-1de3-454e-84d1-f0eba4ecda25-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:45:19.845483 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.845407    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5be9f52b-ccab-4249-890c-be03403ee233" path="/var/lib/kubelet/pods/5be9f52b-ccab-4249-890c-be03403ee233/volumes"
Aug 04 03:45:19.845796 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:19.845783    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f390fc39-1de3-454e-84d1-f0eba4ecda25" path="/var/lib/kubelet/pods/f390fc39-1de3-454e-84d1-f0eba4ecda25/volumes"
Aug 04 03:45:23.083359 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.083267    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"]
Aug 04 03:45:23.083855 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.083837    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f390fc39-1de3-454e-84d1-f0eba4ecda25" containerName="maas-api"
Aug 04 03:45:23.083942 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.083859    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f390fc39-1de3-454e-84d1-f0eba4ecda25" containerName="maas-api"
Aug 04 03:45:23.083942 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.083873    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5be9f52b-ccab-4249-890c-be03403ee233" containerName="payload-pre-processing"
Aug 04 03:45:23.083942 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.083881    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5be9f52b-ccab-4249-890c-be03403ee233" containerName="payload-pre-processing"
Aug 04 03:45:23.084110 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.084018    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5be9f52b-ccab-4249-890c-be03403ee233" containerName="payload-pre-processing"
Aug 04 03:45:23.084110 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.084032    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f390fc39-1de3-454e-84d1-f0eba4ecda25" containerName="maas-api"
Aug 04 03:45:23.088761 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.088739    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"
Aug 04 03:45:23.091104 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.091086    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-822464dd-serving-cert\""
Aug 04 03:45:23.095687 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.095660    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"]
Aug 04 03:45:23.129540 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.129508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7527de-f4e9-4444-a3d6-202f52add841-openshift-service-ca\") pod \"maas-api-e2e-migrate-822464dd-dd487f498-8mljw\" (UID: \"be7527de-f4e9-4444-a3d6-202f52add841\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"
Aug 04 03:45:23.129683 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.129562    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99tq7\" (UniqueName: \"kubernetes.io/projected/be7527de-f4e9-4444-a3d6-202f52add841-kube-api-access-99tq7\") pod \"maas-api-e2e-migrate-822464dd-dd487f498-8mljw\" (UID: \"be7527de-f4e9-4444-a3d6-202f52add841\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"
Aug 04 03:45:23.129741 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.129669    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be7527de-f4e9-4444-a3d6-202f52add841-maas-api-tls\") pod \"maas-api-e2e-migrate-822464dd-dd487f498-8mljw\" (UID: \"be7527de-f4e9-4444-a3d6-202f52add841\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"
Aug 04 03:45:23.230881 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.230847    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7527de-f4e9-4444-a3d6-202f52add841-openshift-service-ca\") pod \"maas-api-e2e-migrate-822464dd-dd487f498-8mljw\" (UID: \"be7527de-f4e9-4444-a3d6-202f52add841\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"
Aug 04 03:45:23.231065 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.230901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99tq7\" (UniqueName: \"kubernetes.io/projected/be7527de-f4e9-4444-a3d6-202f52add841-kube-api-access-99tq7\") pod \"maas-api-e2e-migrate-822464dd-dd487f498-8mljw\" (UID: \"be7527de-f4e9-4444-a3d6-202f52add841\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"
Aug 04 03:45:23.231065 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.230951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be7527de-f4e9-4444-a3d6-202f52add841-maas-api-tls\") pod \"maas-api-e2e-migrate-822464dd-dd487f498-8mljw\" (UID: \"be7527de-f4e9-4444-a3d6-202f52add841\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"
Aug 04 03:45:23.231525 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.231502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7527de-f4e9-4444-a3d6-202f52add841-openshift-service-ca\") pod \"maas-api-e2e-migrate-822464dd-dd487f498-8mljw\" (UID: \"be7527de-f4e9-4444-a3d6-202f52add841\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"
Aug 04 03:45:23.233340 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.233317    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be7527de-f4e9-4444-a3d6-202f52add841-maas-api-tls\") pod \"maas-api-e2e-migrate-822464dd-dd487f498-8mljw\" (UID: \"be7527de-f4e9-4444-a3d6-202f52add841\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"
Aug 04 03:45:23.238779 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.238752    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99tq7\" (UniqueName: \"kubernetes.io/projected/be7527de-f4e9-4444-a3d6-202f52add841-kube-api-access-99tq7\") pod \"maas-api-e2e-migrate-822464dd-dd487f498-8mljw\" (UID: \"be7527de-f4e9-4444-a3d6-202f52add841\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"
Aug 04 03:45:23.402456 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.402368    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"
Aug 04 03:45:23.690873 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.690779    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb"]
Aug 04 03:45:23.694486 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.694471    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb"
Aug 04 03:45:23.697015 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.696993    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-822464dd\""
Aug 04 03:45:23.704684 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.704655    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb"]
Aug 04 03:45:23.736596 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.736566    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6abd63-7a08-4f90-bd96-9d458cf6df54-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb\" (UID: \"9d6abd63-7a08-4f90-bd96-9d458cf6df54\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb"
Aug 04 03:45:23.736768 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.736603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhv8v\" (UniqueName: \"kubernetes.io/projected/9d6abd63-7a08-4f90-bd96-9d458cf6df54-kube-api-access-lhv8v\") pod \"payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb\" (UID: \"9d6abd63-7a08-4f90-bd96-9d458cf6df54\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb"
Aug 04 03:45:23.755998 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.755972    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"]
Aug 04 03:45:23.757540 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:45:23.757513    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7527de_f4e9_4444_a3d6_202f52add841.slice/crio-1739f2127a44ea9380246cbd73dd2255c494f05cc4c59e85ccf6d540f9e5e0ae WatchSource:0}: Error finding container 1739f2127a44ea9380246cbd73dd2255c494f05cc4c59e85ccf6d540f9e5e0ae: Status 404 returned error can't find the container with id 1739f2127a44ea9380246cbd73dd2255c494f05cc4c59e85ccf6d540f9e5e0ae
Aug 04 03:45:23.838028 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.837988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6abd63-7a08-4f90-bd96-9d458cf6df54-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb\" (UID: \"9d6abd63-7a08-4f90-bd96-9d458cf6df54\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb"
Aug 04 03:45:23.838198 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.838048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhv8v\" (UniqueName: \"kubernetes.io/projected/9d6abd63-7a08-4f90-bd96-9d458cf6df54-kube-api-access-lhv8v\") pod \"payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb\" (UID: \"9d6abd63-7a08-4f90-bd96-9d458cf6df54\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb"
Aug 04 03:45:23.838709 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.838679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6abd63-7a08-4f90-bd96-9d458cf6df54-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb\" (UID: \"9d6abd63-7a08-4f90-bd96-9d458cf6df54\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb"
Aug 04 03:45:23.846637 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:23.846606    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhv8v\" (UniqueName: \"kubernetes.io/projected/9d6abd63-7a08-4f90-bd96-9d458cf6df54-kube-api-access-lhv8v\") pod \"payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb\" (UID: \"9d6abd63-7a08-4f90-bd96-9d458cf6df54\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb"
Aug 04 03:45:24.006729 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:24.006689    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb"
Aug 04 03:45:24.143047 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:24.143021    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb"]
Aug 04 03:45:24.148255 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:45:24.148227    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6abd63_7a08_4f90_bd96_9d458cf6df54.slice/crio-0682defb875213fcc0339b61701f203acc9a082ddea3846adafaaab78f4a919f WatchSource:0}: Error finding container 0682defb875213fcc0339b61701f203acc9a082ddea3846adafaaab78f4a919f: Status 404 returned error can't find the container with id 0682defb875213fcc0339b61701f203acc9a082ddea3846adafaaab78f4a919f
Aug 04 03:45:24.647384 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:24.647345    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb" event={"ID":"9d6abd63-7a08-4f90-bd96-9d458cf6df54","Type":"ContainerStarted","Data":"f974219bc7b7aeade692bcc48c1cac65ccb12afdee956c113ff5bf73f46b3165"}
Aug 04 03:45:24.647700 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:24.647507    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb"
Aug 04 03:45:24.647700 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:24.647549    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb" event={"ID":"9d6abd63-7a08-4f90-bd96-9d458cf6df54","Type":"ContainerStarted","Data":"0682defb875213fcc0339b61701f203acc9a082ddea3846adafaaab78f4a919f"}
Aug 04 03:45:24.648895 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:24.648870    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw" event={"ID":"be7527de-f4e9-4444-a3d6-202f52add841","Type":"ContainerStarted","Data":"090cc2a583568c437ed7940e556c7e647dc45eb9a838731fbce4d22c84ce0f61"}
Aug 04 03:45:24.649033 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:24.648909    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw" event={"ID":"be7527de-f4e9-4444-a3d6-202f52add841","Type":"ContainerStarted","Data":"1739f2127a44ea9380246cbd73dd2255c494f05cc4c59e85ccf6d540f9e5e0ae"}
Aug 04 03:45:24.649033 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:24.648998    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"
Aug 04 03:45:24.664477 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:24.664434    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb" podStartSLOduration=1.664419563 podStartE2EDuration="1.664419563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:45:24.663360683 +0000 UTC m=+2155.343147809" watchObservedRunningTime="2026-08-04 03:45:24.664419563 +0000 UTC m=+2155.344206681"
Aug 04 03:45:24.681842 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:24.681796    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw" podStartSLOduration=1.328185565 podStartE2EDuration="1.681781698s" totalImagesPullingTime="353.596133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:45:24.68120921 +0000 UTC m=+2155.360996331" watchObservedRunningTime="2026-08-04 03:45:24.681781698 +0000 UTC m=+2155.361568816"
Aug 04 03:45:30.660843 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:30.660802    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"
Aug 04 03:45:35.657312 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:35.657282    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb"
Aug 04 03:45:53.608842 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:53.608805    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"]
Aug 04 03:45:53.609357 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:53.609077    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw" podUID="be7527de-f4e9-4444-a3d6-202f52add841" containerName="maas-api" containerID="cri-o://090cc2a583568c437ed7940e556c7e647dc45eb9a838731fbce4d22c84ce0f61" gracePeriod=30
Aug 04 03:45:53.660727 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:53.660571    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb"]
Aug 04 03:45:53.660917 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:53.660837    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb" podUID="9d6abd63-7a08-4f90-bd96-9d458cf6df54" containerName="payload-pre-processing" containerID="cri-o://f974219bc7b7aeade692bcc48c1cac65ccb12afdee956c113ff5bf73f46b3165" gracePeriod=30
Aug 04 03:45:53.792823 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:53.792782    2579 generic.go:356] "Generic (PLEG): container finished" podID="be7527de-f4e9-4444-a3d6-202f52add841" containerID="090cc2a583568c437ed7940e556c7e647dc45eb9a838731fbce4d22c84ce0f61" exitCode=0
Aug 04 03:45:53.793042 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:53.792847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw" event={"ID":"be7527de-f4e9-4444-a3d6-202f52add841","Type":"ContainerDied","Data":"090cc2a583568c437ed7940e556c7e647dc45eb9a838731fbce4d22c84ce0f61"}
Aug 04 03:45:53.794575 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:53.794551    2579 generic.go:356] "Generic (PLEG): container finished" podID="9d6abd63-7a08-4f90-bd96-9d458cf6df54" containerID="f974219bc7b7aeade692bcc48c1cac65ccb12afdee956c113ff5bf73f46b3165" exitCode=0
Aug 04 03:45:53.794719 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:53.794598    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb" event={"ID":"9d6abd63-7a08-4f90-bd96-9d458cf6df54","Type":"ContainerDied","Data":"f974219bc7b7aeade692bcc48c1cac65ccb12afdee956c113ff5bf73f46b3165"}
Aug 04 03:45:53.922929 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:53.922904    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"
Aug 04 03:45:53.949777 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:53.949753    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb"
Aug 04 03:45:54.020800 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.020762    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7527de-f4e9-4444-a3d6-202f52add841-openshift-service-ca\") pod \"be7527de-f4e9-4444-a3d6-202f52add841\" (UID: \"be7527de-f4e9-4444-a3d6-202f52add841\") "
Aug 04 03:45:54.021024 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.020815    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99tq7\" (UniqueName: \"kubernetes.io/projected/be7527de-f4e9-4444-a3d6-202f52add841-kube-api-access-99tq7\") pod \"be7527de-f4e9-4444-a3d6-202f52add841\" (UID: \"be7527de-f4e9-4444-a3d6-202f52add841\") "
Aug 04 03:45:54.021024 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.020875    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be7527de-f4e9-4444-a3d6-202f52add841-maas-api-tls\") pod \"be7527de-f4e9-4444-a3d6-202f52add841\" (UID: \"be7527de-f4e9-4444-a3d6-202f52add841\") "
Aug 04 03:45:54.021252 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.021218    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7527de-f4e9-4444-a3d6-202f52add841-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be7527de-f4e9-4444-a3d6-202f52add841" (UID: "be7527de-f4e9-4444-a3d6-202f52add841"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:45:54.023091 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.023063    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be7527de-f4e9-4444-a3d6-202f52add841-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be7527de-f4e9-4444-a3d6-202f52add841" (UID: "be7527de-f4e9-4444-a3d6-202f52add841"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:45:54.023209 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.023114    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7527de-f4e9-4444-a3d6-202f52add841-kube-api-access-99tq7" (OuterVolumeSpecName: "kube-api-access-99tq7") pod "be7527de-f4e9-4444-a3d6-202f52add841" (UID: "be7527de-f4e9-4444-a3d6-202f52add841"). InnerVolumeSpecName "kube-api-access-99tq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:45:54.121618 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.121582    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhv8v\" (UniqueName: \"kubernetes.io/projected/9d6abd63-7a08-4f90-bd96-9d458cf6df54-kube-api-access-lhv8v\") pod \"9d6abd63-7a08-4f90-bd96-9d458cf6df54\" (UID: \"9d6abd63-7a08-4f90-bd96-9d458cf6df54\") "
Aug 04 03:45:54.121774 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.121626    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6abd63-7a08-4f90-bd96-9d458cf6df54-plugins-config-volume\") pod \"9d6abd63-7a08-4f90-bd96-9d458cf6df54\" (UID: \"9d6abd63-7a08-4f90-bd96-9d458cf6df54\") "
Aug 04 03:45:54.121833 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.121822    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7527de-f4e9-4444-a3d6-202f52add841-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:45:54.121871 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.121835    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99tq7\" (UniqueName: \"kubernetes.io/projected/be7527de-f4e9-4444-a3d6-202f52add841-kube-api-access-99tq7\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:45:54.121871 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.121846    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be7527de-f4e9-4444-a3d6-202f52add841-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:45:54.122075 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.122042    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d6abd63-7a08-4f90-bd96-9d458cf6df54-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d6abd63-7a08-4f90-bd96-9d458cf6df54" (UID: "9d6abd63-7a08-4f90-bd96-9d458cf6df54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:45:54.123720 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.123696    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6abd63-7a08-4f90-bd96-9d458cf6df54-kube-api-access-lhv8v" (OuterVolumeSpecName: "kube-api-access-lhv8v") pod "9d6abd63-7a08-4f90-bd96-9d458cf6df54" (UID: "9d6abd63-7a08-4f90-bd96-9d458cf6df54"). InnerVolumeSpecName "kube-api-access-lhv8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:45:54.223124 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.223093    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhv8v\" (UniqueName: \"kubernetes.io/projected/9d6abd63-7a08-4f90-bd96-9d458cf6df54-kube-api-access-lhv8v\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:45:54.223124 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.223123    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6abd63-7a08-4f90-bd96-9d458cf6df54-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:45:54.800447 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.800410    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb"
Aug 04 03:45:54.800447 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.800439    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb" event={"ID":"9d6abd63-7a08-4f90-bd96-9d458cf6df54","Type":"ContainerDied","Data":"0682defb875213fcc0339b61701f203acc9a082ddea3846adafaaab78f4a919f"}
Aug 04 03:45:54.801045 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.800499    2579 scope.go:117] "RemoveContainer" containerID="f974219bc7b7aeade692bcc48c1cac65ccb12afdee956c113ff5bf73f46b3165"
Aug 04 03:45:54.801868 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.801849    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw" event={"ID":"be7527de-f4e9-4444-a3d6-202f52add841","Type":"ContainerDied","Data":"1739f2127a44ea9380246cbd73dd2255c494f05cc4c59e85ccf6d540f9e5e0ae"}
Aug 04 03:45:54.801868 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.801865    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"
Aug 04 03:45:54.811578 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.811470    2579 scope.go:117] "RemoveContainer" containerID="090cc2a583568c437ed7940e556c7e647dc45eb9a838731fbce4d22c84ce0f61"
Aug 04 03:45:54.829320 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.829299    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"]
Aug 04 03:45:54.835262 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.835235    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-822464dd-dd487f498-8mljw"]
Aug 04 03:45:54.846268 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.846241    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb"]
Aug 04 03:45:54.849715 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:54.849691    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-822464dd-684fc488d4-9jwpb"]
Aug 04 03:45:55.845564 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:55.845533    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d6abd63-7a08-4f90-bd96-9d458cf6df54" path="/var/lib/kubelet/pods/9d6abd63-7a08-4f90-bd96-9d458cf6df54/volumes"
Aug 04 03:45:55.846051 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:55.845936    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be7527de-f4e9-4444-a3d6-202f52add841" path="/var/lib/kubelet/pods/be7527de-f4e9-4444-a3d6-202f52add841/volumes"
Aug 04 03:45:58.203945 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.203914    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"]
Aug 04 03:45:58.204410 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.204348    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d6abd63-7a08-4f90-bd96-9d458cf6df54" containerName="payload-pre-processing"
Aug 04 03:45:58.204410 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.204361    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d6abd63-7a08-4f90-bd96-9d458cf6df54" containerName="payload-pre-processing"
Aug 04 03:45:58.204410 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.204375    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be7527de-f4e9-4444-a3d6-202f52add841" containerName="maas-api"
Aug 04 03:45:58.204410 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.204381    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="be7527de-f4e9-4444-a3d6-202f52add841" containerName="maas-api"
Aug 04 03:45:58.204617 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.204440    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d6abd63-7a08-4f90-bd96-9d458cf6df54" containerName="payload-pre-processing"
Aug 04 03:45:58.204617 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.204454    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="be7527de-f4e9-4444-a3d6-202f52add841" containerName="maas-api"
Aug 04 03:45:58.209076 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.209057    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"
Aug 04 03:45:58.211772 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.211747    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9e9f9650-serving-cert\""
Aug 04 03:45:58.216134 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.216108    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"]
Aug 04 03:45:58.255348 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.255316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb183dfd-be9a-446a-a550-bdf8fe7c39d1-maas-api-tls\") pod \"maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk\" (UID: \"fb183dfd-be9a-446a-a550-bdf8fe7c39d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"
Aug 04 03:45:58.255533 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.255420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb183dfd-be9a-446a-a550-bdf8fe7c39d1-openshift-service-ca\") pod \"maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk\" (UID: \"fb183dfd-be9a-446a-a550-bdf8fe7c39d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"
Aug 04 03:45:58.255533 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.255464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xwsl\" (UniqueName: \"kubernetes.io/projected/fb183dfd-be9a-446a-a550-bdf8fe7c39d1-kube-api-access-2xwsl\") pod \"maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk\" (UID: \"fb183dfd-be9a-446a-a550-bdf8fe7c39d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"
Aug 04 03:45:58.356213 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.356178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xwsl\" (UniqueName: \"kubernetes.io/projected/fb183dfd-be9a-446a-a550-bdf8fe7c39d1-kube-api-access-2xwsl\") pod \"maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk\" (UID: \"fb183dfd-be9a-446a-a550-bdf8fe7c39d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"
Aug 04 03:45:58.356405 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.356316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb183dfd-be9a-446a-a550-bdf8fe7c39d1-maas-api-tls\") pod \"maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk\" (UID: \"fb183dfd-be9a-446a-a550-bdf8fe7c39d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"
Aug 04 03:45:58.356405 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.356363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb183dfd-be9a-446a-a550-bdf8fe7c39d1-openshift-service-ca\") pod \"maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk\" (UID: \"fb183dfd-be9a-446a-a550-bdf8fe7c39d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"
Aug 04 03:45:58.357007 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.356985    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb183dfd-be9a-446a-a550-bdf8fe7c39d1-openshift-service-ca\") pod \"maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk\" (UID: \"fb183dfd-be9a-446a-a550-bdf8fe7c39d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"
Aug 04 03:45:58.358885 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.358861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb183dfd-be9a-446a-a550-bdf8fe7c39d1-maas-api-tls\") pod \"maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk\" (UID: \"fb183dfd-be9a-446a-a550-bdf8fe7c39d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"
Aug 04 03:45:58.365251 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.365227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xwsl\" (UniqueName: \"kubernetes.io/projected/fb183dfd-be9a-446a-a550-bdf8fe7c39d1-kube-api-access-2xwsl\") pod \"maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk\" (UID: \"fb183dfd-be9a-446a-a550-bdf8fe7c39d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"
Aug 04 03:45:58.525464 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.525427    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"
Aug 04 03:45:58.671280 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.671249    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"]
Aug 04 03:45:58.673060 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:45:58.673032    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb183dfd_be9a_446a_a550_bdf8fe7c39d1.slice/crio-0c7812bd58d64109634f8a6c676229c770235cecde2028137edce074ad4f35b5 WatchSource:0}: Error finding container 0c7812bd58d64109634f8a6c676229c770235cecde2028137edce074ad4f35b5: Status 404 returned error can't find the container with id 0c7812bd58d64109634f8a6c676229c770235cecde2028137edce074ad4f35b5
Aug 04 03:45:58.825322 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.825237    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk" event={"ID":"fb183dfd-be9a-446a-a550-bdf8fe7c39d1","Type":"ContainerStarted","Data":"0c7812bd58d64109634f8a6c676229c770235cecde2028137edce074ad4f35b5"}
Aug 04 03:45:58.840913 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.840872    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq"]
Aug 04 03:45:58.845616 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.845590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq"
Aug 04 03:45:58.848453 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.848431    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9e9f9650-dockercfg-4hcrq\""
Aug 04 03:45:58.850305 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.850284    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9e9f9650\""
Aug 04 03:45:58.859017 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.858994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fblx\" (UniqueName: \"kubernetes.io/projected/e1339c49-f710-4a5f-84df-cce4f6659d17-kube-api-access-8fblx\") pod \"payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq\" (UID: \"e1339c49-f710-4a5f-84df-cce4f6659d17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq"
Aug 04 03:45:58.859120 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.859036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1339c49-f710-4a5f-84df-cce4f6659d17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq\" (UID: \"e1339c49-f710-4a5f-84df-cce4f6659d17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq"
Aug 04 03:45:58.861856 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.861833    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq"]
Aug 04 03:45:58.959528 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.959494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1339c49-f710-4a5f-84df-cce4f6659d17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq\" (UID: \"e1339c49-f710-4a5f-84df-cce4f6659d17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq"
Aug 04 03:45:58.959693 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.959616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fblx\" (UniqueName: \"kubernetes.io/projected/e1339c49-f710-4a5f-84df-cce4f6659d17-kube-api-access-8fblx\") pod \"payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq\" (UID: \"e1339c49-f710-4a5f-84df-cce4f6659d17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq"
Aug 04 03:45:58.960291 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.960266    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1339c49-f710-4a5f-84df-cce4f6659d17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq\" (UID: \"e1339c49-f710-4a5f-84df-cce4f6659d17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq"
Aug 04 03:45:58.968016 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:58.967989    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fblx\" (UniqueName: \"kubernetes.io/projected/e1339c49-f710-4a5f-84df-cce4f6659d17-kube-api-access-8fblx\") pod \"payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq\" (UID: \"e1339c49-f710-4a5f-84df-cce4f6659d17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq"
Aug 04 03:45:59.156643 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:59.156558    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq"
Aug 04 03:45:59.307680 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:59.307647    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq"]
Aug 04 03:45:59.309843 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:45:59.309813    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1339c49_f710_4a5f_84df_cce4f6659d17.slice/crio-47a4520d6fcdcc1846e86ae8645a2601d7992805b69b26dc92ee2255f2ef95af WatchSource:0}: Error finding container 47a4520d6fcdcc1846e86ae8645a2601d7992805b69b26dc92ee2255f2ef95af: Status 404 returned error can't find the container with id 47a4520d6fcdcc1846e86ae8645a2601d7992805b69b26dc92ee2255f2ef95af
Aug 04 03:45:59.831598 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:59.831562    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk" event={"ID":"fb183dfd-be9a-446a-a550-bdf8fe7c39d1","Type":"ContainerStarted","Data":"d016a73439c51f8918ce10cb8f4f72a05f0e34d904821c3fe915b2263a12303a"}
Aug 04 03:45:59.831810 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:59.831674    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"
Aug 04 03:45:59.833272 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:59.833244    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq" event={"ID":"e1339c49-f710-4a5f-84df-cce4f6659d17","Type":"ContainerStarted","Data":"cc3759d110be2c17f93f4982dd2202009dbdb20c227da7b7afa9624079b625f1"}
Aug 04 03:45:59.833390 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:59.833287    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq" event={"ID":"e1339c49-f710-4a5f-84df-cce4f6659d17","Type":"ContainerStarted","Data":"47a4520d6fcdcc1846e86ae8645a2601d7992805b69b26dc92ee2255f2ef95af"}
Aug 04 03:45:59.833390 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:59.833311    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq"
Aug 04 03:45:59.849385 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:59.849333    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk" podStartSLOduration=1.355757658 podStartE2EDuration="1.849313958s" totalImagesPullingTime="493.5563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:45:59.846362302 +0000 UTC m=+2190.526149444" watchObservedRunningTime="2026-08-04 03:45:59.849313958 +0000 UTC m=+2190.529101079"
Aug 04 03:45:59.867916 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:45:59.867849    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq" podStartSLOduration=1.867828076 podStartE2EDuration="1.867828076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:45:59.866247313 +0000 UTC m=+2190.546034452" watchObservedRunningTime="2026-08-04 03:45:59.867828076 +0000 UTC m=+2190.547615197"
Aug 04 03:46:05.846055 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:05.846021    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"
Aug 04 03:46:10.840561 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:10.840533    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq"
Aug 04 03:46:28.871228 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:28.871196    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"]
Aug 04 03:46:28.871843 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:28.871434    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk" podUID="fb183dfd-be9a-446a-a550-bdf8fe7c39d1" containerName="maas-api" containerID="cri-o://d016a73439c51f8918ce10cb8f4f72a05f0e34d904821c3fe915b2263a12303a" gracePeriod=30
Aug 04 03:46:28.932433 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:28.932402    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq"]
Aug 04 03:46:28.932800 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:28.932660    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq" podUID="e1339c49-f710-4a5f-84df-cce4f6659d17" containerName="payload-pre-processing" containerID="cri-o://cc3759d110be2c17f93f4982dd2202009dbdb20c227da7b7afa9624079b625f1" gracePeriod=30
Aug 04 03:46:29.181815 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.181792    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"
Aug 04 03:46:29.210204 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.210177    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq"
Aug 04 03:46:29.348177 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.348142    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xwsl\" (UniqueName: \"kubernetes.io/projected/fb183dfd-be9a-446a-a550-bdf8fe7c39d1-kube-api-access-2xwsl\") pod \"fb183dfd-be9a-446a-a550-bdf8fe7c39d1\" (UID: \"fb183dfd-be9a-446a-a550-bdf8fe7c39d1\") "
Aug 04 03:46:29.348348 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.348204    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb183dfd-be9a-446a-a550-bdf8fe7c39d1-openshift-service-ca\") pod \"fb183dfd-be9a-446a-a550-bdf8fe7c39d1\" (UID: \"fb183dfd-be9a-446a-a550-bdf8fe7c39d1\") "
Aug 04 03:46:29.348348 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.348253    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb183dfd-be9a-446a-a550-bdf8fe7c39d1-maas-api-tls\") pod \"fb183dfd-be9a-446a-a550-bdf8fe7c39d1\" (UID: \"fb183dfd-be9a-446a-a550-bdf8fe7c39d1\") "
Aug 04 03:46:29.348348 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.348276    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1339c49-f710-4a5f-84df-cce4f6659d17-plugins-config-volume\") pod \"e1339c49-f710-4a5f-84df-cce4f6659d17\" (UID: \"e1339c49-f710-4a5f-84df-cce4f6659d17\") "
Aug 04 03:46:29.348348 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.348322    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fblx\" (UniqueName: \"kubernetes.io/projected/e1339c49-f710-4a5f-84df-cce4f6659d17-kube-api-access-8fblx\") pod \"e1339c49-f710-4a5f-84df-cce4f6659d17\" (UID: \"e1339c49-f710-4a5f-84df-cce4f6659d17\") "
Aug 04 03:46:29.348530 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.348507    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb183dfd-be9a-446a-a550-bdf8fe7c39d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb183dfd-be9a-446a-a550-bdf8fe7c39d1" (UID: "fb183dfd-be9a-446a-a550-bdf8fe7c39d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:46:29.348664 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.348638    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb183dfd-be9a-446a-a550-bdf8fe7c39d1-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:46:29.348775 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.348690    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1339c49-f710-4a5f-84df-cce4f6659d17-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e1339c49-f710-4a5f-84df-cce4f6659d17" (UID: "e1339c49-f710-4a5f-84df-cce4f6659d17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:46:29.350416 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.350395    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb183dfd-be9a-446a-a550-bdf8fe7c39d1-kube-api-access-2xwsl" (OuterVolumeSpecName: "kube-api-access-2xwsl") pod "fb183dfd-be9a-446a-a550-bdf8fe7c39d1" (UID: "fb183dfd-be9a-446a-a550-bdf8fe7c39d1"). InnerVolumeSpecName "kube-api-access-2xwsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:46:29.350524 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.350503    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb183dfd-be9a-446a-a550-bdf8fe7c39d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb183dfd-be9a-446a-a550-bdf8fe7c39d1" (UID: "fb183dfd-be9a-446a-a550-bdf8fe7c39d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:46:29.350589 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.350565    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1339c49-f710-4a5f-84df-cce4f6659d17-kube-api-access-8fblx" (OuterVolumeSpecName: "kube-api-access-8fblx") pod "e1339c49-f710-4a5f-84df-cce4f6659d17" (UID: "e1339c49-f710-4a5f-84df-cce4f6659d17"). InnerVolumeSpecName "kube-api-access-8fblx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:46:29.449917 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.449834    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xwsl\" (UniqueName: \"kubernetes.io/projected/fb183dfd-be9a-446a-a550-bdf8fe7c39d1-kube-api-access-2xwsl\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:46:29.449917 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.449870    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb183dfd-be9a-446a-a550-bdf8fe7c39d1-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:46:29.449917 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.449883    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1339c49-f710-4a5f-84df-cce4f6659d17-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:46:29.449917 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.449896    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fblx\" (UniqueName: \"kubernetes.io/projected/e1339c49-f710-4a5f-84df-cce4f6659d17-kube-api-access-8fblx\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:46:29.983948 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.983909    2579 generic.go:356] "Generic (PLEG): container finished" podID="fb183dfd-be9a-446a-a550-bdf8fe7c39d1" containerID="d016a73439c51f8918ce10cb8f4f72a05f0e34d904821c3fe915b2263a12303a" exitCode=0
Aug 04 03:46:29.984394 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.983979    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"
Aug 04 03:46:29.984394 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.983997    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk" event={"ID":"fb183dfd-be9a-446a-a550-bdf8fe7c39d1","Type":"ContainerDied","Data":"d016a73439c51f8918ce10cb8f4f72a05f0e34d904821c3fe915b2263a12303a"}
Aug 04 03:46:29.984394 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.984061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk" event={"ID":"fb183dfd-be9a-446a-a550-bdf8fe7c39d1","Type":"ContainerDied","Data":"0c7812bd58d64109634f8a6c676229c770235cecde2028137edce074ad4f35b5"}
Aug 04 03:46:29.984394 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.984092    2579 scope.go:117] "RemoveContainer" containerID="d016a73439c51f8918ce10cb8f4f72a05f0e34d904821c3fe915b2263a12303a"
Aug 04 03:46:29.985593 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.985567    2579 generic.go:356] "Generic (PLEG): container finished" podID="e1339c49-f710-4a5f-84df-cce4f6659d17" containerID="cc3759d110be2c17f93f4982dd2202009dbdb20c227da7b7afa9624079b625f1" exitCode=0
Aug 04 03:46:29.985702 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.985675    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq" event={"ID":"e1339c49-f710-4a5f-84df-cce4f6659d17","Type":"ContainerDied","Data":"cc3759d110be2c17f93f4982dd2202009dbdb20c227da7b7afa9624079b625f1"}
Aug 04 03:46:29.985769 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.985719    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq" event={"ID":"e1339c49-f710-4a5f-84df-cce4f6659d17","Type":"ContainerDied","Data":"47a4520d6fcdcc1846e86ae8645a2601d7992805b69b26dc92ee2255f2ef95af"}
Aug 04 03:46:29.985834 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.985784    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq"
Aug 04 03:46:29.993421 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.993400    2579 scope.go:117] "RemoveContainer" containerID="d016a73439c51f8918ce10cb8f4f72a05f0e34d904821c3fe915b2263a12303a"
Aug 04 03:46:29.993755 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:46:29.993728    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d016a73439c51f8918ce10cb8f4f72a05f0e34d904821c3fe915b2263a12303a\": container with ID starting with d016a73439c51f8918ce10cb8f4f72a05f0e34d904821c3fe915b2263a12303a not found: ID does not exist" containerID="d016a73439c51f8918ce10cb8f4f72a05f0e34d904821c3fe915b2263a12303a"
Aug 04 03:46:29.993842 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.993767    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d016a73439c51f8918ce10cb8f4f72a05f0e34d904821c3fe915b2263a12303a"} err="failed to get container status \"d016a73439c51f8918ce10cb8f4f72a05f0e34d904821c3fe915b2263a12303a\": rpc error: code = NotFound desc = could not find container \"d016a73439c51f8918ce10cb8f4f72a05f0e34d904821c3fe915b2263a12303a\": container with ID starting with d016a73439c51f8918ce10cb8f4f72a05f0e34d904821c3fe915b2263a12303a not found: ID does not exist"
Aug 04 03:46:29.993842 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:29.993793    2579 scope.go:117] "RemoveContainer" containerID="cc3759d110be2c17f93f4982dd2202009dbdb20c227da7b7afa9624079b625f1"
Aug 04 03:46:30.001292 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:30.001273    2579 scope.go:117] "RemoveContainer" containerID="cc3759d110be2c17f93f4982dd2202009dbdb20c227da7b7afa9624079b625f1"
Aug 04 03:46:30.001535 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:46:30.001516    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3759d110be2c17f93f4982dd2202009dbdb20c227da7b7afa9624079b625f1\": container with ID starting with cc3759d110be2c17f93f4982dd2202009dbdb20c227da7b7afa9624079b625f1 not found: ID does not exist" containerID="cc3759d110be2c17f93f4982dd2202009dbdb20c227da7b7afa9624079b625f1"
Aug 04 03:46:30.001608 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:30.001541    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3759d110be2c17f93f4982dd2202009dbdb20c227da7b7afa9624079b625f1"} err="failed to get container status \"cc3759d110be2c17f93f4982dd2202009dbdb20c227da7b7afa9624079b625f1\": rpc error: code = NotFound desc = could not find container \"cc3759d110be2c17f93f4982dd2202009dbdb20c227da7b7afa9624079b625f1\": container with ID starting with cc3759d110be2c17f93f4982dd2202009dbdb20c227da7b7afa9624079b625f1 not found: ID does not exist"
Aug 04 03:46:30.003188 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:30.003166    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq"]
Aug 04 03:46:30.006581 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:30.006558    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9e9f9650-6cbcb79b4d-l7nmq"]
Aug 04 03:46:30.016794 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:30.016768    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"]
Aug 04 03:46:30.020697 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:30.020673    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e9f9650-668c987fd-9ldkk"]
Aug 04 03:46:31.844767 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:31.844732    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1339c49-f710-4a5f-84df-cce4f6659d17" path="/var/lib/kubelet/pods/e1339c49-f710-4a5f-84df-cce4f6659d17/volumes"
Aug 04 03:46:31.845193 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:31.845164    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb183dfd-be9a-446a-a550-bdf8fe7c39d1" path="/var/lib/kubelet/pods/fb183dfd-be9a-446a-a550-bdf8fe7c39d1/volumes"
Aug 04 03:46:34.442767 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.442733    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"]
Aug 04 03:46:34.443483 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.443455    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1339c49-f710-4a5f-84df-cce4f6659d17" containerName="payload-pre-processing"
Aug 04 03:46:34.443483 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.443482    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1339c49-f710-4a5f-84df-cce4f6659d17" containerName="payload-pre-processing"
Aug 04 03:46:34.443677 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.443517    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb183dfd-be9a-446a-a550-bdf8fe7c39d1" containerName="maas-api"
Aug 04 03:46:34.443677 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.443528    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb183dfd-be9a-446a-a550-bdf8fe7c39d1" containerName="maas-api"
Aug 04 03:46:34.443677 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.443631    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1339c49-f710-4a5f-84df-cce4f6659d17" containerName="payload-pre-processing"
Aug 04 03:46:34.443677 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.443648    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb183dfd-be9a-446a-a550-bdf8fe7c39d1" containerName="maas-api"
Aug 04 03:46:34.448348 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.448326    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"
Aug 04 03:46:34.450780 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.450757    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-69e40c6b-serving-cert\""
Aug 04 03:46:34.459356 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.459330    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"]
Aug 04 03:46:34.600954 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.600917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b032e1e-1655-4637-b186-a0a36adfffd2-openshift-service-ca\") pod \"maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9\" (UID: \"8b032e1e-1655-4637-b186-a0a36adfffd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"
Aug 04 03:46:34.601151 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.600981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28t9p\" (UniqueName: \"kubernetes.io/projected/8b032e1e-1655-4637-b186-a0a36adfffd2-kube-api-access-28t9p\") pod \"maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9\" (UID: \"8b032e1e-1655-4637-b186-a0a36adfffd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"
Aug 04 03:46:34.601151 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.601041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b032e1e-1655-4637-b186-a0a36adfffd2-maas-api-tls\") pod \"maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9\" (UID: \"8b032e1e-1655-4637-b186-a0a36adfffd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"
Aug 04 03:46:34.701944 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.701852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b032e1e-1655-4637-b186-a0a36adfffd2-openshift-service-ca\") pod \"maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9\" (UID: \"8b032e1e-1655-4637-b186-a0a36adfffd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"
Aug 04 03:46:34.701944 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.701896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28t9p\" (UniqueName: \"kubernetes.io/projected/8b032e1e-1655-4637-b186-a0a36adfffd2-kube-api-access-28t9p\") pod \"maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9\" (UID: \"8b032e1e-1655-4637-b186-a0a36adfffd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"
Aug 04 03:46:34.702204 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.702001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b032e1e-1655-4637-b186-a0a36adfffd2-maas-api-tls\") pod \"maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9\" (UID: \"8b032e1e-1655-4637-b186-a0a36adfffd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"
Aug 04 03:46:34.702565 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.702545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b032e1e-1655-4637-b186-a0a36adfffd2-openshift-service-ca\") pod \"maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9\" (UID: \"8b032e1e-1655-4637-b186-a0a36adfffd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"
Aug 04 03:46:34.704388 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.704368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b032e1e-1655-4637-b186-a0a36adfffd2-maas-api-tls\") pod \"maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9\" (UID: \"8b032e1e-1655-4637-b186-a0a36adfffd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"
Aug 04 03:46:34.710566 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.710547    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28t9p\" (UniqueName: \"kubernetes.io/projected/8b032e1e-1655-4637-b186-a0a36adfffd2-kube-api-access-28t9p\") pod \"maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9\" (UID: \"8b032e1e-1655-4637-b186-a0a36adfffd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"
Aug 04 03:46:34.761192 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.761163    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"
Aug 04 03:46:34.898181 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:34.898153    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"]
Aug 04 03:46:34.899885 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:46:34.899860    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b032e1e_1655_4637_b186_a0a36adfffd2.slice/crio-78c3a26de80680001b3f2e9784b979dd7e18521198612f370a2935f1b0021c85 WatchSource:0}: Error finding container 78c3a26de80680001b3f2e9784b979dd7e18521198612f370a2935f1b0021c85: Status 404 returned error can't find the container with id 78c3a26de80680001b3f2e9784b979dd7e18521198612f370a2935f1b0021c85
Aug 04 03:46:35.014677 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.014637    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9" event={"ID":"8b032e1e-1655-4637-b186-a0a36adfffd2","Type":"ContainerStarted","Data":"78c3a26de80680001b3f2e9784b979dd7e18521198612f370a2935f1b0021c85"}
Aug 04 03:46:35.072409 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.072381    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr"]
Aug 04 03:46:35.077049 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.077028    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr"
Aug 04 03:46:35.080593 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.080566    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-69e40c6b\""
Aug 04 03:46:35.081101 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.081086    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-69e40c6b-dockercfg-vfm5v\""
Aug 04 03:46:35.098141 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.098116    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr"]
Aug 04 03:46:35.112952 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.112929    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w"]
Aug 04 03:46:35.117021 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.117001    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w"
Aug 04 03:46:35.126264 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.126238    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w"]
Aug 04 03:46:35.206200 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.206155    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz87s\" (UniqueName: \"kubernetes.io/projected/498c0591-4230-470b-bd93-85e57d9a5d24-kube-api-access-wz87s\") pod \"payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr\" (UID: \"498c0591-4230-470b-bd93-85e57d9a5d24\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr"
Aug 04 03:46:35.206200 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.206197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/498c0591-4230-470b-bd93-85e57d9a5d24-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr\" (UID: \"498c0591-4230-470b-bd93-85e57d9a5d24\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr"
Aug 04 03:46:35.307280 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.307179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g64v2\" (UniqueName: \"kubernetes.io/projected/23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc-kube-api-access-g64v2\") pod \"payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w\" (UID: \"23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc\") " pod="openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w"
Aug 04 03:46:35.307280 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.307246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz87s\" (UniqueName: \"kubernetes.io/projected/498c0591-4230-470b-bd93-85e57d9a5d24-kube-api-access-wz87s\") pod \"payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr\" (UID: \"498c0591-4230-470b-bd93-85e57d9a5d24\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr"
Aug 04 03:46:35.307513 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.307282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/498c0591-4230-470b-bd93-85e57d9a5d24-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr\" (UID: \"498c0591-4230-470b-bd93-85e57d9a5d24\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr"
Aug 04 03:46:35.307513 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.307363    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc-plugins-config-volume\") pod \"payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w\" (UID: \"23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc\") " pod="openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w"
Aug 04 03:46:35.308046 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.308025    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/498c0591-4230-470b-bd93-85e57d9a5d24-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr\" (UID: \"498c0591-4230-470b-bd93-85e57d9a5d24\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr"
Aug 04 03:46:35.315424 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.315403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz87s\" (UniqueName: \"kubernetes.io/projected/498c0591-4230-470b-bd93-85e57d9a5d24-kube-api-access-wz87s\") pod \"payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr\" (UID: \"498c0591-4230-470b-bd93-85e57d9a5d24\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr"
Aug 04 03:46:35.387404 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.387375    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr"
Aug 04 03:46:35.407803 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.407777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g64v2\" (UniqueName: \"kubernetes.io/projected/23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc-kube-api-access-g64v2\") pod \"payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w\" (UID: \"23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc\") " pod="openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w"
Aug 04 03:46:35.407921 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.407864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc-plugins-config-volume\") pod \"payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w\" (UID: \"23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc\") " pod="openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w"
Aug 04 03:46:35.408662 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.408642    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc-plugins-config-volume\") pod \"payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w\" (UID: \"23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc\") " pod="openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w"
Aug 04 03:46:35.417593 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.417564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g64v2\" (UniqueName: \"kubernetes.io/projected/23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc-kube-api-access-g64v2\") pod \"payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w\" (UID: \"23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc\") " pod="openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w"
Aug 04 03:46:35.428822 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.428795    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w"
Aug 04 03:46:35.539055 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.539020    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr"]
Aug 04 03:46:35.540808 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:46:35.540688    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498c0591_4230_470b_bd93_85e57d9a5d24.slice/crio-0082cce1bbddf31caba0b8a1774c2b93ea9cf952223506e632ade6c0243c27e8 WatchSource:0}: Error finding container 0082cce1bbddf31caba0b8a1774c2b93ea9cf952223506e632ade6c0243c27e8: Status 404 returned error can't find the container with id 0082cce1bbddf31caba0b8a1774c2b93ea9cf952223506e632ade6c0243c27e8
Aug 04 03:46:35.578681 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:35.578654    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w"]
Aug 04 03:46:35.582221 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:46:35.582184    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ee2460_9fd3_42ef_84ed_e8cb1ecbc9fc.slice/crio-660b63918cd8a5ebea67c80641d7f9d5421c838a65bf1c4de19913f131002ddc WatchSource:0}: Error finding container 660b63918cd8a5ebea67c80641d7f9d5421c838a65bf1c4de19913f131002ddc: Status 404 returned error can't find the container with id 660b63918cd8a5ebea67c80641d7f9d5421c838a65bf1c4de19913f131002ddc
Aug 04 03:46:36.020757 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:36.020724    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w" event={"ID":"23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc","Type":"ContainerStarted","Data":"37b272f6e6798d70a49c92ad20c09eea3b8b3f402b3e7824c4f8ab4e4bddebba"}
Aug 04 03:46:36.020973 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:36.020781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w" event={"ID":"23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc","Type":"ContainerStarted","Data":"660b63918cd8a5ebea67c80641d7f9d5421c838a65bf1c4de19913f131002ddc"}
Aug 04 03:46:36.020973 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:36.020839    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w"
Aug 04 03:46:36.022299 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:36.022273    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr" event={"ID":"498c0591-4230-470b-bd93-85e57d9a5d24","Type":"ContainerStarted","Data":"3faad5598696e8e9830689b24969b26d886361df176c58f002482ecf91b6628f"}
Aug 04 03:46:36.022430 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:36.022321    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr" event={"ID":"498c0591-4230-470b-bd93-85e57d9a5d24","Type":"ContainerStarted","Data":"0082cce1bbddf31caba0b8a1774c2b93ea9cf952223506e632ade6c0243c27e8"}
Aug 04 03:46:36.022430 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:36.022346    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr"
Aug 04 03:46:36.023977 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:36.023935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9" event={"ID":"8b032e1e-1655-4637-b186-a0a36adfffd2","Type":"ContainerStarted","Data":"3e57ac09c2d787aa1f32a8d95a97b3f3004eaf0ba5a41f19a030b2263309d72b"}
Aug 04 03:46:36.024161 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:36.024142    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"
Aug 04 03:46:36.037536 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:36.037492    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w" podStartSLOduration=1.037479491 podStartE2EDuration="1.037479491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:46:36.036466735 +0000 UTC m=+2226.716253854" watchObservedRunningTime="2026-08-04 03:46:36.037479491 +0000 UTC m=+2226.717266611"
Aug 04 03:46:36.053503 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:36.053458    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9" podStartSLOduration=1.585439472 podStartE2EDuration="2.05344435s" totalImagesPullingTime="468.004878ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:46:36.051665071 +0000 UTC m=+2226.731452194" watchObservedRunningTime="2026-08-04 03:46:36.05344435 +0000 UTC m=+2226.733231468"
Aug 04 03:46:36.066245 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:36.066207    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr" podStartSLOduration=1.066195759 podStartE2EDuration="1.066195759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:46:36.065573827 +0000 UTC m=+2226.745360947" watchObservedRunningTime="2026-08-04 03:46:36.066195759 +0000 UTC m=+2226.745982878"
Aug 04 03:46:42.034433 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:42.034400    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"
Aug 04 03:46:47.030567 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:47.030537    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w"
Aug 04 03:46:47.030985 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:46:47.030624    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr"
Aug 04 03:47:04.883407 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:04.883372    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"]
Aug 04 03:47:04.884022 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:04.883662    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9" podUID="8b032e1e-1655-4637-b186-a0a36adfffd2" containerName="maas-api" containerID="cri-o://3e57ac09c2d787aa1f32a8d95a97b3f3004eaf0ba5a41f19a030b2263309d72b" gracePeriod=30
Aug 04 03:47:04.931721 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:04.931688    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w"]
Aug 04 03:47:04.931952 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:04.931924    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w" podUID="23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc" containerName="payload-processing" containerID="cri-o://37b272f6e6798d70a49c92ad20c09eea3b8b3f402b3e7824c4f8ab4e4bddebba" gracePeriod=30
Aug 04 03:47:04.939604 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:04.939198    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr"]
Aug 04 03:47:04.939604 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:04.939423    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr" podUID="498c0591-4230-470b-bd93-85e57d9a5d24" containerName="payload-pre-processing" containerID="cri-o://3faad5598696e8e9830689b24969b26d886361df176c58f002482ecf91b6628f" gracePeriod=30
Aug 04 03:47:05.175683 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.175649    2579 generic.go:356] "Generic (PLEG): container finished" podID="8b032e1e-1655-4637-b186-a0a36adfffd2" containerID="3e57ac09c2d787aa1f32a8d95a97b3f3004eaf0ba5a41f19a030b2263309d72b" exitCode=0
Aug 04 03:47:05.176555 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.175779    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9" event={"ID":"8b032e1e-1655-4637-b186-a0a36adfffd2","Type":"ContainerDied","Data":"3e57ac09c2d787aa1f32a8d95a97b3f3004eaf0ba5a41f19a030b2263309d72b"}
Aug 04 03:47:05.178731 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.178707    2579 generic.go:356] "Generic (PLEG): container finished" podID="23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc" containerID="37b272f6e6798d70a49c92ad20c09eea3b8b3f402b3e7824c4f8ab4e4bddebba" exitCode=0
Aug 04 03:47:05.178885 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.178785    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w" event={"ID":"23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc","Type":"ContainerDied","Data":"37b272f6e6798d70a49c92ad20c09eea3b8b3f402b3e7824c4f8ab4e4bddebba"}
Aug 04 03:47:05.181575 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.181517    2579 generic.go:356] "Generic (PLEG): container finished" podID="498c0591-4230-470b-bd93-85e57d9a5d24" containerID="3faad5598696e8e9830689b24969b26d886361df176c58f002482ecf91b6628f" exitCode=0
Aug 04 03:47:05.181701 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.181590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr" event={"ID":"498c0591-4230-470b-bd93-85e57d9a5d24","Type":"ContainerDied","Data":"3faad5598696e8e9830689b24969b26d886361df176c58f002482ecf91b6628f"}
Aug 04 03:47:05.209029 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.209002    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"
Aug 04 03:47:05.227850 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.227821    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr"
Aug 04 03:47:05.239291 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.239271    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w"
Aug 04 03:47:05.291322 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.291290    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b032e1e-1655-4637-b186-a0a36adfffd2-maas-api-tls\") pod \"8b032e1e-1655-4637-b186-a0a36adfffd2\" (UID: \"8b032e1e-1655-4637-b186-a0a36adfffd2\") "
Aug 04 03:47:05.291505 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.291435    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b032e1e-1655-4637-b186-a0a36adfffd2-openshift-service-ca\") pod \"8b032e1e-1655-4637-b186-a0a36adfffd2\" (UID: \"8b032e1e-1655-4637-b186-a0a36adfffd2\") "
Aug 04 03:47:05.291505 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.291487    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28t9p\" (UniqueName: \"kubernetes.io/projected/8b032e1e-1655-4637-b186-a0a36adfffd2-kube-api-access-28t9p\") pod \"8b032e1e-1655-4637-b186-a0a36adfffd2\" (UID: \"8b032e1e-1655-4637-b186-a0a36adfffd2\") "
Aug 04 03:47:05.291780 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.291759    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b032e1e-1655-4637-b186-a0a36adfffd2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8b032e1e-1655-4637-b186-a0a36adfffd2" (UID: "8b032e1e-1655-4637-b186-a0a36adfffd2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:47:05.293358 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.293338    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b032e1e-1655-4637-b186-a0a36adfffd2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8b032e1e-1655-4637-b186-a0a36adfffd2" (UID: "8b032e1e-1655-4637-b186-a0a36adfffd2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:47:05.293469 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.293423    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b032e1e-1655-4637-b186-a0a36adfffd2-kube-api-access-28t9p" (OuterVolumeSpecName: "kube-api-access-28t9p") pod "8b032e1e-1655-4637-b186-a0a36adfffd2" (UID: "8b032e1e-1655-4637-b186-a0a36adfffd2"). InnerVolumeSpecName "kube-api-access-28t9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:47:05.392724 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.392692    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wz87s\" (UniqueName: \"kubernetes.io/projected/498c0591-4230-470b-bd93-85e57d9a5d24-kube-api-access-wz87s\") pod \"498c0591-4230-470b-bd93-85e57d9a5d24\" (UID: \"498c0591-4230-470b-bd93-85e57d9a5d24\") "
Aug 04 03:47:05.392878 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.392747    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/498c0591-4230-470b-bd93-85e57d9a5d24-plugins-config-volume\") pod \"498c0591-4230-470b-bd93-85e57d9a5d24\" (UID: \"498c0591-4230-470b-bd93-85e57d9a5d24\") "
Aug 04 03:47:05.392878 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.392766    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g64v2\" (UniqueName: \"kubernetes.io/projected/23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc-kube-api-access-g64v2\") pod \"23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc\" (UID: \"23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc\") "
Aug 04 03:47:05.392878 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.392797    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc-plugins-config-volume\") pod \"23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc\" (UID: \"23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc\") "
Aug 04 03:47:05.393069 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.393052    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b032e1e-1655-4637-b186-a0a36adfffd2-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:47:05.393069 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.393067    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28t9p\" (UniqueName: \"kubernetes.io/projected/8b032e1e-1655-4637-b186-a0a36adfffd2-kube-api-access-28t9p\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:47:05.393180 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.393077    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b032e1e-1655-4637-b186-a0a36adfffd2-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:47:05.393254 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.393118    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/498c0591-4230-470b-bd93-85e57d9a5d24-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "498c0591-4230-470b-bd93-85e57d9a5d24" (UID: "498c0591-4230-470b-bd93-85e57d9a5d24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:47:05.393336 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.393312    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc" (UID: "23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:47:05.394951 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.394899    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/498c0591-4230-470b-bd93-85e57d9a5d24-kube-api-access-wz87s" (OuterVolumeSpecName: "kube-api-access-wz87s") pod "498c0591-4230-470b-bd93-85e57d9a5d24" (UID: "498c0591-4230-470b-bd93-85e57d9a5d24"). InnerVolumeSpecName "kube-api-access-wz87s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:47:05.394951 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.394904    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc-kube-api-access-g64v2" (OuterVolumeSpecName: "kube-api-access-g64v2") pod "23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc" (UID: "23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc"). InnerVolumeSpecName "kube-api-access-g64v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:47:05.493540 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.493501    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz87s\" (UniqueName: \"kubernetes.io/projected/498c0591-4230-470b-bd93-85e57d9a5d24-kube-api-access-wz87s\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:47:05.493540 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.493530    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/498c0591-4230-470b-bd93-85e57d9a5d24-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:47:05.493540 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.493540    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g64v2\" (UniqueName: \"kubernetes.io/projected/23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc-kube-api-access-g64v2\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:47:05.493540 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:05.493550    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:47:06.190273 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:06.188189    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w" event={"ID":"23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc","Type":"ContainerDied","Data":"660b63918cd8a5ebea67c80641d7f9d5421c838a65bf1c4de19913f131002ddc"}
Aug 04 03:47:06.190273 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:06.188261    2579 scope.go:117] "RemoveContainer" containerID="37b272f6e6798d70a49c92ad20c09eea3b8b3f402b3e7824c4f8ab4e4bddebba"
Aug 04 03:47:06.190273 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:06.188402    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w"
Aug 04 03:47:06.191725 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:06.191692    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr"
Aug 04 03:47:06.191878 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:06.191849    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr" event={"ID":"498c0591-4230-470b-bd93-85e57d9a5d24","Type":"ContainerDied","Data":"0082cce1bbddf31caba0b8a1774c2b93ea9cf952223506e632ade6c0243c27e8"}
Aug 04 03:47:06.201319 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:06.201289    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9" event={"ID":"8b032e1e-1655-4637-b186-a0a36adfffd2","Type":"ContainerDied","Data":"78c3a26de80680001b3f2e9784b979dd7e18521198612f370a2935f1b0021c85"}
Aug 04 03:47:06.201464 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:06.201360    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"
Aug 04 03:47:06.206332 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:06.206306    2579 scope.go:117] "RemoveContainer" containerID="3faad5598696e8e9830689b24969b26d886361df176c58f002482ecf91b6628f"
Aug 04 03:47:06.214752 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:06.214735    2579 scope.go:117] "RemoveContainer" containerID="3e57ac09c2d787aa1f32a8d95a97b3f3004eaf0ba5a41f19a030b2263309d72b"
Aug 04 03:47:06.221341 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:06.221318    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr"]
Aug 04 03:47:06.225233 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:06.225212    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-69e40c6b-5f7cff7848-2lpbr"]
Aug 04 03:47:06.238989 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:06.238954    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w"]
Aug 04 03:47:06.240815 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:06.240795    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-69e40c6b-5b965577bb-nbh4w"]
Aug 04 03:47:06.251655 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:06.251635    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"]
Aug 04 03:47:06.255523 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:06.255507    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-69e40c6b-666c644cb7-qnth9"]
Aug 04 03:47:07.845436 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:07.845403    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc" path="/var/lib/kubelet/pods/23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc/volumes"
Aug 04 03:47:07.845784 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:07.845748    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="498c0591-4230-470b-bd93-85e57d9a5d24" path="/var/lib/kubelet/pods/498c0591-4230-470b-bd93-85e57d9a5d24/volumes"
Aug 04 03:47:07.846103 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:47:07.846090    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b032e1e-1655-4637-b186-a0a36adfffd2" path="/var/lib/kubelet/pods/8b032e1e-1655-4637-b186-a0a36adfffd2/volumes"
Aug 04 03:49:15.794647 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:15.794610    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"]
Aug 04 03:49:15.795272 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:15.795192    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b032e1e-1655-4637-b186-a0a36adfffd2" containerName="maas-api"
Aug 04 03:49:15.795272 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:15.795207    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b032e1e-1655-4637-b186-a0a36adfffd2" containerName="maas-api"
Aug 04 03:49:15.795272 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:15.795241    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc" containerName="payload-processing"
Aug 04 03:49:15.795272 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:15.795250    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc" containerName="payload-processing"
Aug 04 03:49:15.795272 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:15.795266    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="498c0591-4230-470b-bd93-85e57d9a5d24" containerName="payload-pre-processing"
Aug 04 03:49:15.795272 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:15.795275    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="498c0591-4230-470b-bd93-85e57d9a5d24" containerName="payload-pre-processing"
Aug 04 03:49:15.795595 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:15.795358    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ee2460-9fd3-42ef-84ed-e8cb1ecbc9fc" containerName="payload-processing"
Aug 04 03:49:15.795595 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:15.795370    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b032e1e-1655-4637-b186-a0a36adfffd2" containerName="maas-api"
Aug 04 03:49:15.795595 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:15.795380    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="498c0591-4230-470b-bd93-85e57d9a5d24" containerName="payload-pre-processing"
Aug 04 03:49:15.798936 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:15.798912    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"
Aug 04 03:49:15.801536 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:15.801512    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-bb6bbf1b-serving-cert\""
Aug 04 03:49:15.807624 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:15.807405    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"]
Aug 04 03:49:15.904587 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:15.904554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2322def3-897d-41d7-8fc5-80d6790bba54-maas-api-tls\") pod \"maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26\" (UID: \"2322def3-897d-41d7-8fc5-80d6790bba54\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"
Aug 04 03:49:15.904774 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:15.904626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2322def3-897d-41d7-8fc5-80d6790bba54-openshift-service-ca\") pod \"maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26\" (UID: \"2322def3-897d-41d7-8fc5-80d6790bba54\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"
Aug 04 03:49:15.904774 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:15.904667    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbqj5\" (UniqueName: \"kubernetes.io/projected/2322def3-897d-41d7-8fc5-80d6790bba54-kube-api-access-qbqj5\") pod \"maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26\" (UID: \"2322def3-897d-41d7-8fc5-80d6790bba54\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"
Aug 04 03:49:16.005474 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:16.005435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2322def3-897d-41d7-8fc5-80d6790bba54-maas-api-tls\") pod \"maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26\" (UID: \"2322def3-897d-41d7-8fc5-80d6790bba54\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"
Aug 04 03:49:16.005669 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:49:16.005570    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-serving-cert: secret "maas-api-e2e-mt-bb6bbf1b-serving-cert" not found
Aug 04 03:49:16.005669 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:16.005607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2322def3-897d-41d7-8fc5-80d6790bba54-openshift-service-ca\") pod \"maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26\" (UID: \"2322def3-897d-41d7-8fc5-80d6790bba54\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"
Aug 04 03:49:16.005669 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:49:16.005644    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2322def3-897d-41d7-8fc5-80d6790bba54-maas-api-tls podName:2322def3-897d-41d7-8fc5-80d6790bba54 nodeName:}" failed. No retries permitted until 2026-08-04 03:49:16.505624571 +0000 UTC m=+2387.185411682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2322def3-897d-41d7-8fc5-80d6790bba54-maas-api-tls") pod "maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26" (UID: "2322def3-897d-41d7-8fc5-80d6790bba54") : secret "maas-api-e2e-mt-bb6bbf1b-serving-cert" not found
Aug 04 03:49:16.005861 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:16.005711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbqj5\" (UniqueName: \"kubernetes.io/projected/2322def3-897d-41d7-8fc5-80d6790bba54-kube-api-access-qbqj5\") pod \"maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26\" (UID: \"2322def3-897d-41d7-8fc5-80d6790bba54\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"
Aug 04 03:49:16.006319 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:16.006298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2322def3-897d-41d7-8fc5-80d6790bba54-openshift-service-ca\") pod \"maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26\" (UID: \"2322def3-897d-41d7-8fc5-80d6790bba54\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"
Aug 04 03:49:16.016602 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:16.016574    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbqj5\" (UniqueName: \"kubernetes.io/projected/2322def3-897d-41d7-8fc5-80d6790bba54-kube-api-access-qbqj5\") pod \"maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26\" (UID: \"2322def3-897d-41d7-8fc5-80d6790bba54\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"
Aug 04 03:49:16.510212 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:16.510160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2322def3-897d-41d7-8fc5-80d6790bba54-maas-api-tls\") pod \"maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26\" (UID: \"2322def3-897d-41d7-8fc5-80d6790bba54\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"
Aug 04 03:49:16.512607 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:16.512583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2322def3-897d-41d7-8fc5-80d6790bba54-maas-api-tls\") pod \"maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26\" (UID: \"2322def3-897d-41d7-8fc5-80d6790bba54\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"
Aug 04 03:49:16.712150 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:16.712110    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"
Aug 04 03:49:16.846382 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:16.846361    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"]
Aug 04 03:49:16.848626 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:49:16.848597    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2322def3_897d_41d7_8fc5_80d6790bba54.slice/crio-f7d0747e38aa2cd0836a9b41a465b374290889c5023ccadce49cea9c24ac2f04 WatchSource:0}: Error finding container f7d0747e38aa2cd0836a9b41a465b374290889c5023ccadce49cea9c24ac2f04: Status 404 returned error can't find the container with id f7d0747e38aa2cd0836a9b41a465b374290889c5023ccadce49cea9c24ac2f04
Aug 04 03:49:17.848413 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:17.848381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26" event={"ID":"2322def3-897d-41d7-8fc5-80d6790bba54","Type":"ContainerStarted","Data":"91f9b323c26697bd5f31fb9c39db70b6164b1e8ea336c7b0c56985d874440270"}
Aug 04 03:49:17.848823 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:17.848429    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26" event={"ID":"2322def3-897d-41d7-8fc5-80d6790bba54","Type":"ContainerStarted","Data":"f7d0747e38aa2cd0836a9b41a465b374290889c5023ccadce49cea9c24ac2f04"}
Aug 04 03:49:17.848823 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:17.848516    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"
Aug 04 03:49:17.863329 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:17.863276    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26" podStartSLOduration=2.420125155 podStartE2EDuration="2.863258408s" totalImagesPullingTime="443.133253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:49:17.863128792 +0000 UTC m=+2388.542915938" watchObservedRunningTime="2026-08-04 03:49:17.863258408 +0000 UTC m=+2388.543045528"
Aug 04 03:49:23.858004 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:23.857977    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"
Aug 04 03:49:46.920061 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:46.920027    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb6bbf1b-ng8q5"]
Aug 04 03:49:46.923511 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:46.923495    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb6bbf1b-ng8q5"
Aug 04 03:49:46.926088 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:46.926061    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7lrf4\""
Aug 04 03:49:46.930181 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:46.930157    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb6bbf1b-ng8q5"]
Aug 04 03:49:46.996541 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:46.996510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4883cad3-10ce-428e-b604-7fe6bef21e76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bb6bbf1b-ng8q5\" (UID: \"4883cad3-10ce-428e-b604-7fe6bef21e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb6bbf1b-ng8q5"
Aug 04 03:49:47.097651 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:47.097603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4883cad3-10ce-428e-b604-7fe6bef21e76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bb6bbf1b-ng8q5\" (UID: \"4883cad3-10ce-428e-b604-7fe6bef21e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb6bbf1b-ng8q5"
Aug 04 03:49:47.098323 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:47.098299    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4883cad3-10ce-428e-b604-7fe6bef21e76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bb6bbf1b-ng8q5\" (UID: \"4883cad3-10ce-428e-b604-7fe6bef21e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb6bbf1b-ng8q5"
Aug 04 03:49:47.236280 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:47.236250    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb6bbf1b-ng8q5"
Aug 04 03:49:47.374313 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:47.374288    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb6bbf1b-ng8q5"]
Aug 04 03:49:47.377400 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:49:47.377371    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4883cad3_10ce_428e_b604_7fe6bef21e76.slice/crio-0b69ad803ad00996896b0d9865f4f6d1322bb9c08bb44a83efbb8e13f7401d37 WatchSource:0}: Error finding container 0b69ad803ad00996896b0d9865f4f6d1322bb9c08bb44a83efbb8e13f7401d37: Status 404 returned error can't find the container with id 0b69ad803ad00996896b0d9865f4f6d1322bb9c08bb44a83efbb8e13f7401d37
Aug 04 03:49:47.994036 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:47.993999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb6bbf1b-ng8q5" event={"ID":"4883cad3-10ce-428e-b604-7fe6bef21e76","Type":"ContainerStarted","Data":"0b69ad803ad00996896b0d9865f4f6d1322bb9c08bb44a83efbb8e13f7401d37"}
Aug 04 03:49:51.013926 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:51.013893    2579 generic.go:356] "Generic (PLEG): container finished" podID="4883cad3-10ce-428e-b604-7fe6bef21e76" containerID="b114bfb233a3f60b53c320de17445c03916f11e54692051e52439acaa356ebfb" exitCode=0
Aug 04 03:49:51.014355 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:51.013989    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb6bbf1b-ng8q5" event={"ID":"4883cad3-10ce-428e-b604-7fe6bef21e76","Type":"ContainerDied","Data":"b114bfb233a3f60b53c320de17445c03916f11e54692051e52439acaa356ebfb"}
Aug 04 03:49:52.150257 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:52.150226    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb6bbf1b-ng8q5"
Aug 04 03:49:52.245935 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:52.245907    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4883cad3-10ce-428e-b604-7fe6bef21e76-maas-api-service-ca\") pod \"4883cad3-10ce-428e-b604-7fe6bef21e76\" (UID: \"4883cad3-10ce-428e-b604-7fe6bef21e76\") "
Aug 04 03:49:52.246250 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:52.246224    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4883cad3-10ce-428e-b604-7fe6bef21e76-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4883cad3-10ce-428e-b604-7fe6bef21e76" (UID: "4883cad3-10ce-428e-b604-7fe6bef21e76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:49:52.347826 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:52.347728    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4883cad3-10ce-428e-b604-7fe6bef21e76-maas-api-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:49:53.026195 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:53.026167    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb6bbf1b-ng8q5"
Aug 04 03:49:53.026369 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:53.026194    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb6bbf1b-ng8q5" event={"ID":"4883cad3-10ce-428e-b604-7fe6bef21e76","Type":"ContainerDied","Data":"0b69ad803ad00996896b0d9865f4f6d1322bb9c08bb44a83efbb8e13f7401d37"}
Aug 04 03:49:53.026369 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:53.026242    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b69ad803ad00996896b0d9865f4f6d1322bb9c08bb44a83efbb8e13f7401d37"
Aug 04 03:49:57.390694 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:57.390662    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"]
Aug 04 03:49:57.391176 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:57.390893    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26" podUID="2322def3-897d-41d7-8fc5-80d6790bba54" containerName="maas-api" containerID="cri-o://91f9b323c26697bd5f31fb9c39db70b6164b1e8ea336c7b0c56985d874440270" gracePeriod=30
Aug 04 03:49:57.567596 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:57.567534    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb6bbf1b-ng8q5"]
Aug 04 03:49:57.570791 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:57.570759    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb6bbf1b-ng8q5"]
Aug 04 03:49:57.643643 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:57.643589    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"
Aug 04 03:49:57.803847 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:57.803815    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2322def3-897d-41d7-8fc5-80d6790bba54-maas-api-tls\") pod \"2322def3-897d-41d7-8fc5-80d6790bba54\" (UID: \"2322def3-897d-41d7-8fc5-80d6790bba54\") "
Aug 04 03:49:57.804044 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:57.803867    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2322def3-897d-41d7-8fc5-80d6790bba54-openshift-service-ca\") pod \"2322def3-897d-41d7-8fc5-80d6790bba54\" (UID: \"2322def3-897d-41d7-8fc5-80d6790bba54\") "
Aug 04 03:49:57.804044 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:57.803929    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbqj5\" (UniqueName: \"kubernetes.io/projected/2322def3-897d-41d7-8fc5-80d6790bba54-kube-api-access-qbqj5\") pod \"2322def3-897d-41d7-8fc5-80d6790bba54\" (UID: \"2322def3-897d-41d7-8fc5-80d6790bba54\") "
Aug 04 03:49:57.804248 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:57.804227    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2322def3-897d-41d7-8fc5-80d6790bba54-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2322def3-897d-41d7-8fc5-80d6790bba54" (UID: "2322def3-897d-41d7-8fc5-80d6790bba54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:49:57.806014 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:57.805983    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2322def3-897d-41d7-8fc5-80d6790bba54-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2322def3-897d-41d7-8fc5-80d6790bba54" (UID: "2322def3-897d-41d7-8fc5-80d6790bba54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:49:57.806120 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:57.806061    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2322def3-897d-41d7-8fc5-80d6790bba54-kube-api-access-qbqj5" (OuterVolumeSpecName: "kube-api-access-qbqj5") pod "2322def3-897d-41d7-8fc5-80d6790bba54" (UID: "2322def3-897d-41d7-8fc5-80d6790bba54"). InnerVolumeSpecName "kube-api-access-qbqj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:49:57.845303 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:57.845267    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4883cad3-10ce-428e-b604-7fe6bef21e76" path="/var/lib/kubelet/pods/4883cad3-10ce-428e-b604-7fe6bef21e76/volumes"
Aug 04 03:49:57.904777 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:57.904702    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbqj5\" (UniqueName: \"kubernetes.io/projected/2322def3-897d-41d7-8fc5-80d6790bba54-kube-api-access-qbqj5\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:49:57.904777 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:57.904727    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2322def3-897d-41d7-8fc5-80d6790bba54-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:49:57.904777 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:57.904738    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2322def3-897d-41d7-8fc5-80d6790bba54-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:49:58.053174 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:58.053141    2579 generic.go:356] "Generic (PLEG): container finished" podID="2322def3-897d-41d7-8fc5-80d6790bba54" containerID="91f9b323c26697bd5f31fb9c39db70b6164b1e8ea336c7b0c56985d874440270" exitCode=0
Aug 04 03:49:58.053374 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:58.053186    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"
Aug 04 03:49:58.053374 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:58.053230    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26" event={"ID":"2322def3-897d-41d7-8fc5-80d6790bba54","Type":"ContainerDied","Data":"91f9b323c26697bd5f31fb9c39db70b6164b1e8ea336c7b0c56985d874440270"}
Aug 04 03:49:58.053374 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:58.053287    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26" event={"ID":"2322def3-897d-41d7-8fc5-80d6790bba54","Type":"ContainerDied","Data":"f7d0747e38aa2cd0836a9b41a465b374290889c5023ccadce49cea9c24ac2f04"}
Aug 04 03:49:58.053374 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:58.053319    2579 scope.go:117] "RemoveContainer" containerID="91f9b323c26697bd5f31fb9c39db70b6164b1e8ea336c7b0c56985d874440270"
Aug 04 03:49:58.061804 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:58.061788    2579 scope.go:117] "RemoveContainer" containerID="91f9b323c26697bd5f31fb9c39db70b6164b1e8ea336c7b0c56985d874440270"
Aug 04 03:49:58.062103 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:49:58.062084    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91f9b323c26697bd5f31fb9c39db70b6164b1e8ea336c7b0c56985d874440270\": container with ID starting with 91f9b323c26697bd5f31fb9c39db70b6164b1e8ea336c7b0c56985d874440270 not found: ID does not exist" containerID="91f9b323c26697bd5f31fb9c39db70b6164b1e8ea336c7b0c56985d874440270"
Aug 04 03:49:58.062159 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:58.062112    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f9b323c26697bd5f31fb9c39db70b6164b1e8ea336c7b0c56985d874440270"} err="failed to get container status \"91f9b323c26697bd5f31fb9c39db70b6164b1e8ea336c7b0c56985d874440270\": rpc error: code = NotFound desc = could not find container \"91f9b323c26697bd5f31fb9c39db70b6164b1e8ea336c7b0c56985d874440270\": container with ID starting with 91f9b323c26697bd5f31fb9c39db70b6164b1e8ea336c7b0c56985d874440270 not found: ID does not exist"
Aug 04 03:49:58.089455 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:58.089429    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"]
Aug 04 03:49:58.104983 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:58.103339    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bb6bbf1b-5676b84647-bhs26"]
Aug 04 03:49:59.844709 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:49:59.844676    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2322def3-897d-41d7-8fc5-80d6790bba54" path="/var/lib/kubelet/pods/2322def3-897d-41d7-8fc5-80d6790bba54/volumes"
Aug 04 03:52:22.764307 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:22.764264    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"]
Aug 04 03:52:22.764899 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:22.764879    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2322def3-897d-41d7-8fc5-80d6790bba54" containerName="maas-api"
Aug 04 03:52:22.765016 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:22.764902    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2322def3-897d-41d7-8fc5-80d6790bba54" containerName="maas-api"
Aug 04 03:52:22.765016 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:22.764924    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4883cad3-10ce-428e-b604-7fe6bef21e76" containerName="revoke-keys"
Aug 04 03:52:22.765016 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:22.764932    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4883cad3-10ce-428e-b604-7fe6bef21e76" containerName="revoke-keys"
Aug 04 03:52:22.765178 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:22.765058    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4883cad3-10ce-428e-b604-7fe6bef21e76" containerName="revoke-keys"
Aug 04 03:52:22.765178 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:22.765074    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2322def3-897d-41d7-8fc5-80d6790bba54" containerName="maas-api"
Aug 04 03:52:22.768229 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:22.768201    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"
Aug 04 03:52:22.770628 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:22.770605    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ea8d5546-serving-cert\""
Aug 04 03:52:22.783025 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:22.783001    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"]
Aug 04 03:52:22.845949 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:22.845919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c08ed69d-f9d6-432e-8c43-df79c1126c34-openshift-service-ca\") pod \"maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p\" (UID: \"c08ed69d-f9d6-432e-8c43-df79c1126c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"
Aug 04 03:52:22.846142 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:22.845953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbgjm\" (UniqueName: \"kubernetes.io/projected/c08ed69d-f9d6-432e-8c43-df79c1126c34-kube-api-access-gbgjm\") pod \"maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p\" (UID: \"c08ed69d-f9d6-432e-8c43-df79c1126c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"
Aug 04 03:52:22.846142 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:22.846009    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c08ed69d-f9d6-432e-8c43-df79c1126c34-maas-api-tls\") pod \"maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p\" (UID: \"c08ed69d-f9d6-432e-8c43-df79c1126c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"
Aug 04 03:52:22.948039 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:22.948002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c08ed69d-f9d6-432e-8c43-df79c1126c34-openshift-service-ca\") pod \"maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p\" (UID: \"c08ed69d-f9d6-432e-8c43-df79c1126c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"
Aug 04 03:52:22.948306 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:22.948282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbgjm\" (UniqueName: \"kubernetes.io/projected/c08ed69d-f9d6-432e-8c43-df79c1126c34-kube-api-access-gbgjm\") pod \"maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p\" (UID: \"c08ed69d-f9d6-432e-8c43-df79c1126c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"
Aug 04 03:52:22.948448 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:22.948340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c08ed69d-f9d6-432e-8c43-df79c1126c34-maas-api-tls\") pod \"maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p\" (UID: \"c08ed69d-f9d6-432e-8c43-df79c1126c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"
Aug 04 03:52:22.948521 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:52:22.948454    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-serving-cert: secret "maas-api-e2e-mt-ea8d5546-serving-cert" not found
Aug 04 03:52:22.948568 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:52:22.948545    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c08ed69d-f9d6-432e-8c43-df79c1126c34-maas-api-tls podName:c08ed69d-f9d6-432e-8c43-df79c1126c34 nodeName:}" failed. No retries permitted until 2026-08-04 03:52:23.448525299 +0000 UTC m=+2574.128312404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c08ed69d-f9d6-432e-8c43-df79c1126c34-maas-api-tls") pod "maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p" (UID: "c08ed69d-f9d6-432e-8c43-df79c1126c34") : secret "maas-api-e2e-mt-ea8d5546-serving-cert" not found
Aug 04 03:52:22.948736 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:22.948705    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c08ed69d-f9d6-432e-8c43-df79c1126c34-openshift-service-ca\") pod \"maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p\" (UID: \"c08ed69d-f9d6-432e-8c43-df79c1126c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"
Aug 04 03:52:22.969363 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:22.969335    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbgjm\" (UniqueName: \"kubernetes.io/projected/c08ed69d-f9d6-432e-8c43-df79c1126c34-kube-api-access-gbgjm\") pod \"maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p\" (UID: \"c08ed69d-f9d6-432e-8c43-df79c1126c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"
Aug 04 03:52:23.425089 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:23.425056    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j"]
Aug 04 03:52:23.428774 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:23.428751    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j"
Aug 04 03:52:23.431586 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:23.431567    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ea8d5546-dockercfg-m8n9z\""
Aug 04 03:52:23.431729 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:23.431591    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ea8d5546\""
Aug 04 03:52:23.437610 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:23.437587    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j"]
Aug 04 03:52:23.453598 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:23.453567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c08ed69d-f9d6-432e-8c43-df79c1126c34-maas-api-tls\") pod \"maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p\" (UID: \"c08ed69d-f9d6-432e-8c43-df79c1126c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"
Aug 04 03:52:23.455885 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:23.455865    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c08ed69d-f9d6-432e-8c43-df79c1126c34-maas-api-tls\") pod \"maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p\" (UID: \"c08ed69d-f9d6-432e-8c43-df79c1126c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"
Aug 04 03:52:23.554676 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:23.554638    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm5dd\" (UniqueName: \"kubernetes.io/projected/b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b-kube-api-access-wm5dd\") pod \"payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j\" (UID: \"b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b\") " pod="openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j"
Aug 04 03:52:23.554832 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:23.554681    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b-plugins-config-volume\") pod \"payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j\" (UID: \"b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b\") " pod="openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j"
Aug 04 03:52:23.655789 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:23.655757    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm5dd\" (UniqueName: \"kubernetes.io/projected/b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b-kube-api-access-wm5dd\") pod \"payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j\" (UID: \"b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b\") " pod="openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j"
Aug 04 03:52:23.655789 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:23.655794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b-plugins-config-volume\") pod \"payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j\" (UID: \"b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b\") " pod="openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j"
Aug 04 03:52:23.656457 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:23.656438    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b-plugins-config-volume\") pod \"payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j\" (UID: \"b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b\") " pod="openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j"
Aug 04 03:52:23.664228 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:23.664197    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm5dd\" (UniqueName: \"kubernetes.io/projected/b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b-kube-api-access-wm5dd\") pod \"payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j\" (UID: \"b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b\") " pod="openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j"
Aug 04 03:52:23.678970 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:23.678891    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"
Aug 04 03:52:23.739943 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:23.739913    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j"
Aug 04 03:52:23.823142 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:23.823115    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"]
Aug 04 03:52:23.825311 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:52:23.825286    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08ed69d_f9d6_432e_8c43_df79c1126c34.slice/crio-2f67de5b2baf075bd14cdabd4c3953a66f836e9fe1967d4222db00b608ac63f8 WatchSource:0}: Error finding container 2f67de5b2baf075bd14cdabd4c3953a66f836e9fe1967d4222db00b608ac63f8: Status 404 returned error can't find the container with id 2f67de5b2baf075bd14cdabd4c3953a66f836e9fe1967d4222db00b608ac63f8
Aug 04 03:52:23.826656 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:23.826636    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 03:52:23.890950 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:23.890926    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j"]
Aug 04 03:52:23.892872 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:52:23.892847    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2dc1b82_1b2f_42b2_ac11_f5dd8c882d2b.slice/crio-643c9270c096150d3ff70f40c87ead8f2675b14b405536a332af64a8ac1a6fc5 WatchSource:0}: Error finding container 643c9270c096150d3ff70f40c87ead8f2675b14b405536a332af64a8ac1a6fc5: Status 404 returned error can't find the container with id 643c9270c096150d3ff70f40c87ead8f2675b14b405536a332af64a8ac1a6fc5
Aug 04 03:52:24.798184 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:24.798150    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p" event={"ID":"c08ed69d-f9d6-432e-8c43-df79c1126c34","Type":"ContainerStarted","Data":"ec180d3979938f111f366e1f96d0a2fd04224b20294f37cdd81d08e63ed43771"}
Aug 04 03:52:24.798386 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:24.798210    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p" event={"ID":"c08ed69d-f9d6-432e-8c43-df79c1126c34","Type":"ContainerStarted","Data":"2f67de5b2baf075bd14cdabd4c3953a66f836e9fe1967d4222db00b608ac63f8"}
Aug 04 03:52:24.798386 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:24.798246    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"
Aug 04 03:52:24.799732 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:24.799705    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j" event={"ID":"b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b","Type":"ContainerStarted","Data":"97cdc5e009d7001908bd60fbaf8a440db4e3b909ffa9db7910a7d68b966f98f2"}
Aug 04 03:52:24.799860 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:24.799750    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j" event={"ID":"b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b","Type":"ContainerStarted","Data":"643c9270c096150d3ff70f40c87ead8f2675b14b405536a332af64a8ac1a6fc5"}
Aug 04 03:52:24.799860 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:24.799827    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j"
Aug 04 03:52:24.817803 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:24.817753    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p" podStartSLOduration=2.331239407 podStartE2EDuration="2.817737858s" totalImagesPullingTime="486.498451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:52:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:52:24.815275875 +0000 UTC m=+2575.495063004" watchObservedRunningTime="2026-08-04 03:52:24.817737858 +0000 UTC m=+2575.497524977"
Aug 04 03:52:24.831950 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:24.831907    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j" podStartSLOduration=1.831892768 podStartE2EDuration="1.831892768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:52:24.830343795 +0000 UTC m=+2575.510130892" watchObservedRunningTime="2026-08-04 03:52:24.831892768 +0000 UTC m=+2575.511679886"
Aug 04 03:52:30.811703 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:30.811669    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"
Aug 04 03:52:35.808025 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:52:35.807991    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j"
Aug 04 03:53:31.272481 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.272441    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"]
Aug 04 03:53:31.273081 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.272720    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p" podUID="c08ed69d-f9d6-432e-8c43-df79c1126c34" containerName="maas-api" containerID="cri-o://ec180d3979938f111f366e1f96d0a2fd04224b20294f37cdd81d08e63ed43771" gracePeriod=30
Aug 04 03:53:31.383535 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.383498    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j"]
Aug 04 03:53:31.383933 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.383753    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j" podUID="b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b" containerName="payload-processing" containerID="cri-o://97cdc5e009d7001908bd60fbaf8a440db4e3b909ffa9db7910a7d68b966f98f2" gracePeriod=30
Aug 04 03:53:31.541018 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.540930    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"
Aug 04 03:53:31.609668 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.609637    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c08ed69d-f9d6-432e-8c43-df79c1126c34-openshift-service-ca\") pod \"c08ed69d-f9d6-432e-8c43-df79c1126c34\" (UID: \"c08ed69d-f9d6-432e-8c43-df79c1126c34\") "
Aug 04 03:53:31.609668 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.609674    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbgjm\" (UniqueName: \"kubernetes.io/projected/c08ed69d-f9d6-432e-8c43-df79c1126c34-kube-api-access-gbgjm\") pod \"c08ed69d-f9d6-432e-8c43-df79c1126c34\" (UID: \"c08ed69d-f9d6-432e-8c43-df79c1126c34\") "
Aug 04 03:53:31.609890 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.609763    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c08ed69d-f9d6-432e-8c43-df79c1126c34-maas-api-tls\") pod \"c08ed69d-f9d6-432e-8c43-df79c1126c34\" (UID: \"c08ed69d-f9d6-432e-8c43-df79c1126c34\") "
Aug 04 03:53:31.610088 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.610059    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08ed69d-f9d6-432e-8c43-df79c1126c34-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c08ed69d-f9d6-432e-8c43-df79c1126c34" (UID: "c08ed69d-f9d6-432e-8c43-df79c1126c34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:53:31.612015 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.611986    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08ed69d-f9d6-432e-8c43-df79c1126c34-kube-api-access-gbgjm" (OuterVolumeSpecName: "kube-api-access-gbgjm") pod "c08ed69d-f9d6-432e-8c43-df79c1126c34" (UID: "c08ed69d-f9d6-432e-8c43-df79c1126c34"). InnerVolumeSpecName "kube-api-access-gbgjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:53:31.612137 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.612095    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c08ed69d-f9d6-432e-8c43-df79c1126c34-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c08ed69d-f9d6-432e-8c43-df79c1126c34" (UID: "c08ed69d-f9d6-432e-8c43-df79c1126c34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:53:31.640219 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.640193    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j"
Aug 04 03:53:31.710996 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.710931    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm5dd\" (UniqueName: \"kubernetes.io/projected/b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b-kube-api-access-wm5dd\") pod \"b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b\" (UID: \"b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b\") "
Aug 04 03:53:31.711180 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.711023    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b-plugins-config-volume\") pod \"b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b\" (UID: \"b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b\") "
Aug 04 03:53:31.711288 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.711267    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c08ed69d-f9d6-432e-8c43-df79c1126c34-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:53:31.711327 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.711294    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbgjm\" (UniqueName: \"kubernetes.io/projected/c08ed69d-f9d6-432e-8c43-df79c1126c34-kube-api-access-gbgjm\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:53:31.711327 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.711311    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c08ed69d-f9d6-432e-8c43-df79c1126c34-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:53:31.711400 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.711370    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b" (UID: "b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:53:31.713084 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.713057    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b-kube-api-access-wm5dd" (OuterVolumeSpecName: "kube-api-access-wm5dd") pod "b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b" (UID: "b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b"). InnerVolumeSpecName "kube-api-access-wm5dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:53:31.812359 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.812269    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:53:31.812359 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:31.812298    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm5dd\" (UniqueName: \"kubernetes.io/projected/b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b-kube-api-access-wm5dd\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:53:32.139018 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:32.138910    2579 generic.go:356] "Generic (PLEG): container finished" podID="b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b" containerID="97cdc5e009d7001908bd60fbaf8a440db4e3b909ffa9db7910a7d68b966f98f2" exitCode=0
Aug 04 03:53:32.139018 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:32.138982    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j"
Aug 04 03:53:32.139018 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:32.139000    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j" event={"ID":"b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b","Type":"ContainerDied","Data":"97cdc5e009d7001908bd60fbaf8a440db4e3b909ffa9db7910a7d68b966f98f2"}
Aug 04 03:53:32.139243 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:32.139061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j" event={"ID":"b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b","Type":"ContainerDied","Data":"643c9270c096150d3ff70f40c87ead8f2675b14b405536a332af64a8ac1a6fc5"}
Aug 04 03:53:32.139243 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:32.139090    2579 scope.go:117] "RemoveContainer" containerID="97cdc5e009d7001908bd60fbaf8a440db4e3b909ffa9db7910a7d68b966f98f2"
Aug 04 03:53:32.140852 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:32.140825    2579 generic.go:356] "Generic (PLEG): container finished" podID="c08ed69d-f9d6-432e-8c43-df79c1126c34" containerID="ec180d3979938f111f366e1f96d0a2fd04224b20294f37cdd81d08e63ed43771" exitCode=0
Aug 04 03:53:32.141015 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:32.140864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p" event={"ID":"c08ed69d-f9d6-432e-8c43-df79c1126c34","Type":"ContainerDied","Data":"ec180d3979938f111f366e1f96d0a2fd04224b20294f37cdd81d08e63ed43771"}
Aug 04 03:53:32.141015 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:32.140884    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"
Aug 04 03:53:32.141015 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:32.140910    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p" event={"ID":"c08ed69d-f9d6-432e-8c43-df79c1126c34","Type":"ContainerDied","Data":"2f67de5b2baf075bd14cdabd4c3953a66f836e9fe1967d4222db00b608ac63f8"}
Aug 04 03:53:32.148103 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:32.148085    2579 scope.go:117] "RemoveContainer" containerID="97cdc5e009d7001908bd60fbaf8a440db4e3b909ffa9db7910a7d68b966f98f2"
Aug 04 03:53:32.148373 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:53:32.148354    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97cdc5e009d7001908bd60fbaf8a440db4e3b909ffa9db7910a7d68b966f98f2\": container with ID starting with 97cdc5e009d7001908bd60fbaf8a440db4e3b909ffa9db7910a7d68b966f98f2 not found: ID does not exist" containerID="97cdc5e009d7001908bd60fbaf8a440db4e3b909ffa9db7910a7d68b966f98f2"
Aug 04 03:53:32.148434 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:32.148397    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97cdc5e009d7001908bd60fbaf8a440db4e3b909ffa9db7910a7d68b966f98f2"} err="failed to get container status \"97cdc5e009d7001908bd60fbaf8a440db4e3b909ffa9db7910a7d68b966f98f2\": rpc error: code = NotFound desc = could not find container \"97cdc5e009d7001908bd60fbaf8a440db4e3b909ffa9db7910a7d68b966f98f2\": container with ID starting with 97cdc5e009d7001908bd60fbaf8a440db4e3b909ffa9db7910a7d68b966f98f2 not found: ID does not exist"
Aug 04 03:53:32.148434 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:32.148416    2579 scope.go:117] "RemoveContainer" containerID="ec180d3979938f111f366e1f96d0a2fd04224b20294f37cdd81d08e63ed43771"
Aug 04 03:53:32.157545 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:32.157521    2579 scope.go:117] "RemoveContainer" containerID="ec180d3979938f111f366e1f96d0a2fd04224b20294f37cdd81d08e63ed43771"
Aug 04 03:53:32.157890 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:53:32.157854    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec180d3979938f111f366e1f96d0a2fd04224b20294f37cdd81d08e63ed43771\": container with ID starting with ec180d3979938f111f366e1f96d0a2fd04224b20294f37cdd81d08e63ed43771 not found: ID does not exist" containerID="ec180d3979938f111f366e1f96d0a2fd04224b20294f37cdd81d08e63ed43771"
Aug 04 03:53:32.157980 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:32.157898    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec180d3979938f111f366e1f96d0a2fd04224b20294f37cdd81d08e63ed43771"} err="failed to get container status \"ec180d3979938f111f366e1f96d0a2fd04224b20294f37cdd81d08e63ed43771\": rpc error: code = NotFound desc = could not find container \"ec180d3979938f111f366e1f96d0a2fd04224b20294f37cdd81d08e63ed43771\": container with ID starting with ec180d3979938f111f366e1f96d0a2fd04224b20294f37cdd81d08e63ed43771 not found: ID does not exist"
Aug 04 03:53:32.158269 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:32.158250    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j"]
Aug 04 03:53:32.161131 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:32.161108    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ea8d5546-78cfbd984b-c5m2j"]
Aug 04 03:53:32.177660 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:32.172457    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"]
Aug 04 03:53:32.180346 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:32.180315    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea8d5546-758ccbc95d-8rh5p"]
Aug 04 03:53:33.844658 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:33.844626    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b" path="/var/lib/kubelet/pods/b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b/volumes"
Aug 04 03:53:33.845158 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:53:33.845072    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c08ed69d-f9d6-432e-8c43-df79c1126c34" path="/var/lib/kubelet/pods/c08ed69d-f9d6-432e-8c43-df79c1126c34/volumes"
Aug 04 03:55:38.779883 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:38.779847    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"]
Aug 04 03:55:38.782239 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:38.780428    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b" containerName="payload-processing"
Aug 04 03:55:38.782239 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:38.780443    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b" containerName="payload-processing"
Aug 04 03:55:38.782239 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:38.780467    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c08ed69d-f9d6-432e-8c43-df79c1126c34" containerName="maas-api"
Aug 04 03:55:38.782239 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:38.780476    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c08ed69d-f9d6-432e-8c43-df79c1126c34" containerName="maas-api"
Aug 04 03:55:38.782239 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:38.780596    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2dc1b82-1b2f-42b2-ac11-f5dd8c882d2b" containerName="payload-processing"
Aug 04 03:55:38.782239 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:38.780606    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c08ed69d-f9d6-432e-8c43-df79c1126c34" containerName="maas-api"
Aug 04 03:55:38.783930 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:38.783910    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"
Aug 04 03:55:38.786483 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:38.786450    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3a60de-serving-cert\""
Aug 04 03:55:38.798895 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:38.798863    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"]
Aug 04 03:55:38.811705 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:38.811673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1cfa30-e11d-4827-9976-80421e61ea7e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm\" (UID: \"2d1cfa30-e11d-4827-9976-80421e61ea7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"
Aug 04 03:55:38.811877 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:38.811743    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d1cfa30-e11d-4827-9976-80421e61ea7e-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm\" (UID: \"2d1cfa30-e11d-4827-9976-80421e61ea7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"
Aug 04 03:55:38.811877 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:38.811830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5k5z\" (UniqueName: \"kubernetes.io/projected/2d1cfa30-e11d-4827-9976-80421e61ea7e-kube-api-access-h5k5z\") pod \"maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm\" (UID: \"2d1cfa30-e11d-4827-9976-80421e61ea7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"
Aug 04 03:55:38.913300 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:38.913261    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5k5z\" (UniqueName: \"kubernetes.io/projected/2d1cfa30-e11d-4827-9976-80421e61ea7e-kube-api-access-h5k5z\") pod \"maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm\" (UID: \"2d1cfa30-e11d-4827-9976-80421e61ea7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"
Aug 04 03:55:38.913497 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:38.913365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1cfa30-e11d-4827-9976-80421e61ea7e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm\" (UID: \"2d1cfa30-e11d-4827-9976-80421e61ea7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"
Aug 04 03:55:38.913578 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:38.913501    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d1cfa30-e11d-4827-9976-80421e61ea7e-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm\" (UID: \"2d1cfa30-e11d-4827-9976-80421e61ea7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"
Aug 04 03:55:38.913989 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:55:38.913949    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-serving-cert: secret "maas-api-e2e-inf-a-3a60de-serving-cert" not found
Aug 04 03:55:38.914076 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:55:38.914049    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d1cfa30-e11d-4827-9976-80421e61ea7e-maas-api-tls podName:2d1cfa30-e11d-4827-9976-80421e61ea7e nodeName:}" failed. No retries permitted until 2026-08-04 03:55:39.414031474 +0000 UTC m=+2770.093818573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d1cfa30-e11d-4827-9976-80421e61ea7e-maas-api-tls") pod "maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm" (UID: "2d1cfa30-e11d-4827-9976-80421e61ea7e") : secret "maas-api-e2e-inf-a-3a60de-serving-cert" not found
Aug 04 03:55:38.914207 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:38.914188    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1cfa30-e11d-4827-9976-80421e61ea7e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm\" (UID: \"2d1cfa30-e11d-4827-9976-80421e61ea7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"
Aug 04 03:55:38.921351 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:38.921322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5k5z\" (UniqueName: \"kubernetes.io/projected/2d1cfa30-e11d-4827-9976-80421e61ea7e-kube-api-access-h5k5z\") pod \"maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm\" (UID: \"2d1cfa30-e11d-4827-9976-80421e61ea7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"
Aug 04 03:55:39.419417 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:39.419377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d1cfa30-e11d-4827-9976-80421e61ea7e-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm\" (UID: \"2d1cfa30-e11d-4827-9976-80421e61ea7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"
Aug 04 03:55:39.421896 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:39.421827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d1cfa30-e11d-4827-9976-80421e61ea7e-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm\" (UID: \"2d1cfa30-e11d-4827-9976-80421e61ea7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"
Aug 04 03:55:39.682162 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:39.682083    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc"]
Aug 04 03:55:39.685717 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:39.685701    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc"
Aug 04 03:55:39.688238 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:39.688211    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3a60de-dockercfg-zf8p5\""
Aug 04 03:55:39.688351 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:39.688269    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3a60de\""
Aug 04 03:55:39.697630 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:39.697604    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"
Aug 04 03:55:39.698940 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:39.698918    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc"]
Aug 04 03:55:39.720980 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:39.720928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfb2x\" (UniqueName: \"kubernetes.io/projected/f99bf947-b980-413a-82d5-a828eca96122-kube-api-access-hfb2x\") pod \"payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc\" (UID: \"f99bf947-b980-413a-82d5-a828eca96122\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc"
Aug 04 03:55:39.721140 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:39.721108    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f99bf947-b980-413a-82d5-a828eca96122-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc\" (UID: \"f99bf947-b980-413a-82d5-a828eca96122\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc"
Aug 04 03:55:39.822229 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:39.822195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f99bf947-b980-413a-82d5-a828eca96122-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc\" (UID: \"f99bf947-b980-413a-82d5-a828eca96122\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc"
Aug 04 03:55:39.822752 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:39.822301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfb2x\" (UniqueName: \"kubernetes.io/projected/f99bf947-b980-413a-82d5-a828eca96122-kube-api-access-hfb2x\") pod \"payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc\" (UID: \"f99bf947-b980-413a-82d5-a828eca96122\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc"
Aug 04 03:55:39.823185 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:39.823162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f99bf947-b980-413a-82d5-a828eca96122-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc\" (UID: \"f99bf947-b980-413a-82d5-a828eca96122\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc"
Aug 04 03:55:39.833564 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:39.833524    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfb2x\" (UniqueName: \"kubernetes.io/projected/f99bf947-b980-413a-82d5-a828eca96122-kube-api-access-hfb2x\") pod \"payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc\" (UID: \"f99bf947-b980-413a-82d5-a828eca96122\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc"
Aug 04 03:55:39.841819 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:39.841694    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"]
Aug 04 03:55:39.845835 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:55:39.845791    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1cfa30_e11d_4827_9976_80421e61ea7e.slice/crio-f7e1446851c2477d1ed61d204086d3ae381ee2fe2e7b6ced7f6863db9640ccd9 WatchSource:0}: Error finding container f7e1446851c2477d1ed61d204086d3ae381ee2fe2e7b6ced7f6863db9640ccd9: Status 404 returned error can't find the container with id f7e1446851c2477d1ed61d204086d3ae381ee2fe2e7b6ced7f6863db9640ccd9
Aug 04 03:55:39.998864 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:39.998826    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc"
Aug 04 03:55:40.135077 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:40.135047    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc"]
Aug 04 03:55:40.137192 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:55:40.137155    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99bf947_b980_413a_82d5_a828eca96122.slice/crio-de2c9aca988120c2b90b531c9c97e598e1fbd1ea419e18d9754071a1368c8e1e WatchSource:0}: Error finding container de2c9aca988120c2b90b531c9c97e598e1fbd1ea419e18d9754071a1368c8e1e: Status 404 returned error can't find the container with id de2c9aca988120c2b90b531c9c97e598e1fbd1ea419e18d9754071a1368c8e1e
Aug 04 03:55:40.778205 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:40.778173    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc" event={"ID":"f99bf947-b980-413a-82d5-a828eca96122","Type":"ContainerStarted","Data":"5e44b6213eddedd6de57fcbf6d9462c518ffeba1d4e4ff29446c3360707dc032"}
Aug 04 03:55:40.778418 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:40.778232    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc" event={"ID":"f99bf947-b980-413a-82d5-a828eca96122","Type":"ContainerStarted","Data":"de2c9aca988120c2b90b531c9c97e598e1fbd1ea419e18d9754071a1368c8e1e"}
Aug 04 03:55:40.778418 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:40.778330    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc"
Aug 04 03:55:40.779748 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:40.779721    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm" event={"ID":"2d1cfa30-e11d-4827-9976-80421e61ea7e","Type":"ContainerStarted","Data":"db8032979b43156a4ceb0b6969b5f7bec89e4e43d28d888c3cde7a7963c69888"}
Aug 04 03:55:40.779905 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:40.779762    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"
Aug 04 03:55:40.779905 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:40.779779    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm" event={"ID":"2d1cfa30-e11d-4827-9976-80421e61ea7e","Type":"ContainerStarted","Data":"f7e1446851c2477d1ed61d204086d3ae381ee2fe2e7b6ced7f6863db9640ccd9"}
Aug 04 03:55:40.794417 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:40.794371    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc" podStartSLOduration=1.7943563550000001 podStartE2EDuration="1.794356355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:55:40.793535618 +0000 UTC m=+2771.473322786" watchObservedRunningTime="2026-08-04 03:55:40.794356355 +0000 UTC m=+2771.474143452"
Aug 04 03:55:40.809475 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:40.809419    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm" podStartSLOduration=2.43201161 podStartE2EDuration="2.809400057s" totalImagesPullingTime="377.388447ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:55:40.809165995 +0000 UTC m=+2771.488953248" watchObservedRunningTime="2026-08-04 03:55:40.809400057 +0000 UTC m=+2771.489187179"
Aug 04 03:55:46.790029 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:46.790000    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"
Aug 04 03:55:51.786655 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:55:51.786578    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc"
Aug 04 03:56:08.192541 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.192507    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"]
Aug 04 03:56:08.196790 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.196768    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"
Aug 04 03:56:08.199412 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.199391    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e38897-serving-cert\""
Aug 04 03:56:08.209990 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.209952    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"]
Aug 04 03:56:08.291737 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.291702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66ec454e-e5fc-478d-9964-0d01253ab42a-maas-api-tls\") pod \"maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2\" (UID: \"66ec454e-e5fc-478d-9964-0d01253ab42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"
Aug 04 03:56:08.291902 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.291787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ec454e-e5fc-478d-9964-0d01253ab42a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2\" (UID: \"66ec454e-e5fc-478d-9964-0d01253ab42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"
Aug 04 03:56:08.291902 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.291813    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmg7l\" (UniqueName: \"kubernetes.io/projected/66ec454e-e5fc-478d-9964-0d01253ab42a-kube-api-access-mmg7l\") pod \"maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2\" (UID: \"66ec454e-e5fc-478d-9964-0d01253ab42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"
Aug 04 03:56:08.392629 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.392589    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66ec454e-e5fc-478d-9964-0d01253ab42a-maas-api-tls\") pod \"maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2\" (UID: \"66ec454e-e5fc-478d-9964-0d01253ab42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"
Aug 04 03:56:08.392832 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.392686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ec454e-e5fc-478d-9964-0d01253ab42a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2\" (UID: \"66ec454e-e5fc-478d-9964-0d01253ab42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"
Aug 04 03:56:08.392832 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.392720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmg7l\" (UniqueName: \"kubernetes.io/projected/66ec454e-e5fc-478d-9964-0d01253ab42a-kube-api-access-mmg7l\") pod \"maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2\" (UID: \"66ec454e-e5fc-478d-9964-0d01253ab42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"
Aug 04 03:56:08.392832 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:56:08.392739    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-serving-cert: secret "maas-api-e2e-inf-b-e38897-serving-cert" not found
Aug 04 03:56:08.392832 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:56:08.392821    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66ec454e-e5fc-478d-9964-0d01253ab42a-maas-api-tls podName:66ec454e-e5fc-478d-9964-0d01253ab42a nodeName:}" failed. No retries permitted until 2026-08-04 03:56:08.892802347 +0000 UTC m=+2799.572589447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66ec454e-e5fc-478d-9964-0d01253ab42a-maas-api-tls") pod "maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2" (UID: "66ec454e-e5fc-478d-9964-0d01253ab42a") : secret "maas-api-e2e-inf-b-e38897-serving-cert" not found
Aug 04 03:56:08.393357 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.393339    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ec454e-e5fc-478d-9964-0d01253ab42a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2\" (UID: \"66ec454e-e5fc-478d-9964-0d01253ab42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"
Aug 04 03:56:08.402453 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.402421    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmg7l\" (UniqueName: \"kubernetes.io/projected/66ec454e-e5fc-478d-9964-0d01253ab42a-kube-api-access-mmg7l\") pod \"maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2\" (UID: \"66ec454e-e5fc-478d-9964-0d01253ab42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"
Aug 04 03:56:08.838257 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.838222    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h"]
Aug 04 03:56:08.842090 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.842064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h"
Aug 04 03:56:08.845433 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.845402    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e38897\""
Aug 04 03:56:08.845560 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.845446    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e38897-dockercfg-6644j\""
Aug 04 03:56:08.851572 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.851551    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j"]
Aug 04 03:56:08.855383 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.855364    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h"]
Aug 04 03:56:08.855466 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.855443    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j"
Aug 04 03:56:08.865935 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.865904    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j"]
Aug 04 03:56:08.898467 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.898436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66ec454e-e5fc-478d-9964-0d01253ab42a-maas-api-tls\") pod \"maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2\" (UID: \"66ec454e-e5fc-478d-9964-0d01253ab42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"
Aug 04 03:56:08.900821 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.900804    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66ec454e-e5fc-478d-9964-0d01253ab42a-maas-api-tls\") pod \"maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2\" (UID: \"66ec454e-e5fc-478d-9964-0d01253ab42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"
Aug 04 03:56:08.999202 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.999166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7h92\" (UniqueName: \"kubernetes.io/projected/ea58dc82-7669-4a88-94cf-64a4898c61a6-kube-api-access-c7h92\") pod \"payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j\" (UID: \"ea58dc82-7669-4a88-94cf-64a4898c61a6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j"
Aug 04 03:56:08.999202 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.999202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h\" (UID: \"f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h"
Aug 04 03:56:08.999414 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.999300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw9ff\" (UniqueName: \"kubernetes.io/projected/f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad-kube-api-access-bw9ff\") pod \"payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h\" (UID: \"f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h"
Aug 04 03:56:08.999414 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:08.999387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea58dc82-7669-4a88-94cf-64a4898c61a6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j\" (UID: \"ea58dc82-7669-4a88-94cf-64a4898c61a6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j"
Aug 04 03:56:09.100246 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.100159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7h92\" (UniqueName: \"kubernetes.io/projected/ea58dc82-7669-4a88-94cf-64a4898c61a6-kube-api-access-c7h92\") pod \"payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j\" (UID: \"ea58dc82-7669-4a88-94cf-64a4898c61a6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j"
Aug 04 03:56:09.100246 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.100196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h\" (UID: \"f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h"
Aug 04 03:56:09.100246 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.100246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw9ff\" (UniqueName: \"kubernetes.io/projected/f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad-kube-api-access-bw9ff\") pod \"payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h\" (UID: \"f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h"
Aug 04 03:56:09.100525 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.100318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea58dc82-7669-4a88-94cf-64a4898c61a6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j\" (UID: \"ea58dc82-7669-4a88-94cf-64a4898c61a6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j"
Aug 04 03:56:09.100947 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.100924    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h\" (UID: \"f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h"
Aug 04 03:56:09.101040 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.100991    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea58dc82-7669-4a88-94cf-64a4898c61a6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j\" (UID: \"ea58dc82-7669-4a88-94cf-64a4898c61a6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j"
Aug 04 03:56:09.108759 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.108733    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7h92\" (UniqueName: \"kubernetes.io/projected/ea58dc82-7669-4a88-94cf-64a4898c61a6-kube-api-access-c7h92\") pod \"payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j\" (UID: \"ea58dc82-7669-4a88-94cf-64a4898c61a6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j"
Aug 04 03:56:09.108882 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.108785    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw9ff\" (UniqueName: \"kubernetes.io/projected/f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad-kube-api-access-bw9ff\") pod \"payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h\" (UID: \"f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h"
Aug 04 03:56:09.111614 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.111586    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"
Aug 04 03:56:09.154366 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.154311    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h"
Aug 04 03:56:09.167049 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.166665    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j"
Aug 04 03:56:09.277976 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:56:09.277925    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ec454e_e5fc_478d_9964_0d01253ab42a.slice/crio-c6941f807a2ba7a4eff23e74f09911ee3a30bd412439be1ba1cb55c041952830 WatchSource:0}: Error finding container c6941f807a2ba7a4eff23e74f09911ee3a30bd412439be1ba1cb55c041952830: Status 404 returned error can't find the container with id c6941f807a2ba7a4eff23e74f09911ee3a30bd412439be1ba1cb55c041952830
Aug 04 03:56:09.278441 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.278415    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"]
Aug 04 03:56:09.327214 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.327187    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h"]
Aug 04 03:56:09.330752 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:56:09.330722    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e1d7e2_b43d_4fe4_bc6c_5a857a2a5dad.slice/crio-555e3ce7a4f2fd91813700edb7ea12e0258aac7cac7839d9a2b22961958c8b8d WatchSource:0}: Error finding container 555e3ce7a4f2fd91813700edb7ea12e0258aac7cac7839d9a2b22961958c8b8d: Status 404 returned error can't find the container with id 555e3ce7a4f2fd91813700edb7ea12e0258aac7cac7839d9a2b22961958c8b8d
Aug 04 03:56:09.351366 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.351344    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j"]
Aug 04 03:56:09.353139 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:56:09.353113    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea58dc82_7669_4a88_94cf_64a4898c61a6.slice/crio-8d8212e0b1167900f6cc378fac19ee2162e75c79a67410774e11fbb7e57318b0 WatchSource:0}: Error finding container 8d8212e0b1167900f6cc378fac19ee2162e75c79a67410774e11fbb7e57318b0: Status 404 returned error can't find the container with id 8d8212e0b1167900f6cc378fac19ee2162e75c79a67410774e11fbb7e57318b0
Aug 04 03:56:09.943633 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.943596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j" event={"ID":"ea58dc82-7669-4a88-94cf-64a4898c61a6","Type":"ContainerStarted","Data":"6755112caec4e8fa84f2d5198ea2d3e4b6e1d0a9edff9b58b304649187a87f8e"}
Aug 04 03:56:09.943880 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.943861    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j"
Aug 04 03:56:09.944063 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.944043    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j" event={"ID":"ea58dc82-7669-4a88-94cf-64a4898c61a6","Type":"ContainerStarted","Data":"8d8212e0b1167900f6cc378fac19ee2162e75c79a67410774e11fbb7e57318b0"}
Aug 04 03:56:09.946159 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.945569    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2" event={"ID":"66ec454e-e5fc-478d-9964-0d01253ab42a","Type":"ContainerStarted","Data":"4a2921c0d28980feff099b95660bb5c19e1fd9964e5dfd5891870996c0d10891"}
Aug 04 03:56:09.946159 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.945613    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2" event={"ID":"66ec454e-e5fc-478d-9964-0d01253ab42a","Type":"ContainerStarted","Data":"c6941f807a2ba7a4eff23e74f09911ee3a30bd412439be1ba1cb55c041952830"}
Aug 04 03:56:09.946159 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.945652    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"
Aug 04 03:56:09.947643 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.947623    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h" event={"ID":"f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad","Type":"ContainerStarted","Data":"8bd2c0ba922bdcbd4ab4d3cf7d36c1179aa8b17a80afce103a68556a1f3f4e0d"}
Aug 04 03:56:09.947751 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.947656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h" event={"ID":"f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad","Type":"ContainerStarted","Data":"555e3ce7a4f2fd91813700edb7ea12e0258aac7cac7839d9a2b22961958c8b8d"}
Aug 04 03:56:09.947811 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.947770    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h"
Aug 04 03:56:09.962970 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.962903    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j" podStartSLOduration=1.962887915 podStartE2EDuration="1.962887915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:56:09.961158495 +0000 UTC m=+2800.640945638" watchObservedRunningTime="2026-08-04 03:56:09.962887915 +0000 UTC m=+2800.642675012"
Aug 04 03:56:09.980260 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.980214    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2" podStartSLOduration=1.421961718 podStartE2EDuration="1.980198806s" totalImagesPullingTime="558.237088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:56:09.97856568 +0000 UTC m=+2800.658352792" watchObservedRunningTime="2026-08-04 03:56:09.980198806 +0000 UTC m=+2800.659985925"
Aug 04 03:56:09.996748 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:09.996700    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h" podStartSLOduration=1.99668953 podStartE2EDuration="1.99668953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:56:09.99524282 +0000 UTC m=+2800.675029970" watchObservedRunningTime="2026-08-04 03:56:09.99668953 +0000 UTC m=+2800.676476650"
Aug 04 03:56:15.958839 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:15.958812    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"
Aug 04 03:56:20.955566 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:20.955532    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h"
Aug 04 03:56:21.961205 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:21.961120    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j"
Aug 04 03:56:31.103921 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:56:31.103888    2579 scope.go:117] "RemoveContainer" containerID="b114bfb233a3f60b53c320de17445c03916f11e54692051e52439acaa356ebfb"
Aug 04 03:58:18.495486 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:18.495452    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"]
Aug 04 03:58:18.496045 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:18.495693    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm" podUID="2d1cfa30-e11d-4827-9976-80421e61ea7e" containerName="maas-api" containerID="cri-o://db8032979b43156a4ceb0b6969b5f7bec89e4e43d28d888c3cde7a7963c69888" gracePeriod=30
Aug 04 03:58:18.549417 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:18.549383    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc"]
Aug 04 03:58:18.549669 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:18.549630    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc" podUID="f99bf947-b980-413a-82d5-a828eca96122" containerName="payload-processing" containerID="cri-o://5e44b6213eddedd6de57fcbf6d9462c518ffeba1d4e4ff29446c3360707dc032" gracePeriod=30
Aug 04 03:58:18.627066 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:18.627036    2579 generic.go:356] "Generic (PLEG): container finished" podID="2d1cfa30-e11d-4827-9976-80421e61ea7e" containerID="db8032979b43156a4ceb0b6969b5f7bec89e4e43d28d888c3cde7a7963c69888" exitCode=0
Aug 04 03:58:18.627217 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:18.627111    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm" event={"ID":"2d1cfa30-e11d-4827-9976-80421e61ea7e","Type":"ContainerDied","Data":"db8032979b43156a4ceb0b6969b5f7bec89e4e43d28d888c3cde7a7963c69888"}
Aug 04 03:58:18.797561 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:18.797535    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"
Aug 04 03:58:18.832642 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:18.832620    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc"
Aug 04 03:58:18.919216 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:18.919178    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1cfa30-e11d-4827-9976-80421e61ea7e-openshift-service-ca\") pod \"2d1cfa30-e11d-4827-9976-80421e61ea7e\" (UID: \"2d1cfa30-e11d-4827-9976-80421e61ea7e\") "
Aug 04 03:58:18.919216 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:18.919219    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d1cfa30-e11d-4827-9976-80421e61ea7e-maas-api-tls\") pod \"2d1cfa30-e11d-4827-9976-80421e61ea7e\" (UID: \"2d1cfa30-e11d-4827-9976-80421e61ea7e\") "
Aug 04 03:58:18.919431 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:18.919404    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5k5z\" (UniqueName: \"kubernetes.io/projected/2d1cfa30-e11d-4827-9976-80421e61ea7e-kube-api-access-h5k5z\") pod \"2d1cfa30-e11d-4827-9976-80421e61ea7e\" (UID: \"2d1cfa30-e11d-4827-9976-80421e61ea7e\") "
Aug 04 03:58:18.919624 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:18.919597    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1cfa30-e11d-4827-9976-80421e61ea7e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2d1cfa30-e11d-4827-9976-80421e61ea7e" (UID: "2d1cfa30-e11d-4827-9976-80421e61ea7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:58:18.919750 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:18.919733    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1cfa30-e11d-4827-9976-80421e61ea7e-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:58:18.921374 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:18.921347    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1cfa30-e11d-4827-9976-80421e61ea7e-kube-api-access-h5k5z" (OuterVolumeSpecName: "kube-api-access-h5k5z") pod "2d1cfa30-e11d-4827-9976-80421e61ea7e" (UID: "2d1cfa30-e11d-4827-9976-80421e61ea7e"). InnerVolumeSpecName "kube-api-access-h5k5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:58:18.921468 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:18.921417    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d1cfa30-e11d-4827-9976-80421e61ea7e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2d1cfa30-e11d-4827-9976-80421e61ea7e" (UID: "2d1cfa30-e11d-4827-9976-80421e61ea7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:58:19.020120 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.020034    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfb2x\" (UniqueName: \"kubernetes.io/projected/f99bf947-b980-413a-82d5-a828eca96122-kube-api-access-hfb2x\") pod \"f99bf947-b980-413a-82d5-a828eca96122\" (UID: \"f99bf947-b980-413a-82d5-a828eca96122\") "
Aug 04 03:58:19.020120 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.020108    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f99bf947-b980-413a-82d5-a828eca96122-plugins-config-volume\") pod \"f99bf947-b980-413a-82d5-a828eca96122\" (UID: \"f99bf947-b980-413a-82d5-a828eca96122\") "
Aug 04 03:58:19.020359 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.020348    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d1cfa30-e11d-4827-9976-80421e61ea7e-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:58:19.020398 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.020361    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5k5z\" (UniqueName: \"kubernetes.io/projected/2d1cfa30-e11d-4827-9976-80421e61ea7e-kube-api-access-h5k5z\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:58:19.020512 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.020479    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f99bf947-b980-413a-82d5-a828eca96122-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f99bf947-b980-413a-82d5-a828eca96122" (UID: "f99bf947-b980-413a-82d5-a828eca96122"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:58:19.022230 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.022207    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99bf947-b980-413a-82d5-a828eca96122-kube-api-access-hfb2x" (OuterVolumeSpecName: "kube-api-access-hfb2x") pod "f99bf947-b980-413a-82d5-a828eca96122" (UID: "f99bf947-b980-413a-82d5-a828eca96122"). InnerVolumeSpecName "kube-api-access-hfb2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:58:19.121774 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.121739    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f99bf947-b980-413a-82d5-a828eca96122-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:58:19.121774 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.121767    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfb2x\" (UniqueName: \"kubernetes.io/projected/f99bf947-b980-413a-82d5-a828eca96122-kube-api-access-hfb2x\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:58:19.634669 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.634635    2579 generic.go:356] "Generic (PLEG): container finished" podID="f99bf947-b980-413a-82d5-a828eca96122" containerID="5e44b6213eddedd6de57fcbf6d9462c518ffeba1d4e4ff29446c3360707dc032" exitCode=0
Aug 04 03:58:19.635141 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.634683    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc"
Aug 04 03:58:19.635141 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.634722    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc" event={"ID":"f99bf947-b980-413a-82d5-a828eca96122","Type":"ContainerDied","Data":"5e44b6213eddedd6de57fcbf6d9462c518ffeba1d4e4ff29446c3360707dc032"}
Aug 04 03:58:19.635141 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.634783    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc" event={"ID":"f99bf947-b980-413a-82d5-a828eca96122","Type":"ContainerDied","Data":"de2c9aca988120c2b90b531c9c97e598e1fbd1ea419e18d9754071a1368c8e1e"}
Aug 04 03:58:19.635141 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.634817    2579 scope.go:117] "RemoveContainer" containerID="5e44b6213eddedd6de57fcbf6d9462c518ffeba1d4e4ff29446c3360707dc032"
Aug 04 03:58:19.636271 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.636255    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"
Aug 04 03:58:19.636271 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.636260    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm" event={"ID":"2d1cfa30-e11d-4827-9976-80421e61ea7e","Type":"ContainerDied","Data":"f7e1446851c2477d1ed61d204086d3ae381ee2fe2e7b6ced7f6863db9640ccd9"}
Aug 04 03:58:19.645741 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.645722    2579 scope.go:117] "RemoveContainer" containerID="5e44b6213eddedd6de57fcbf6d9462c518ffeba1d4e4ff29446c3360707dc032"
Aug 04 03:58:19.646068 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:58:19.646043    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e44b6213eddedd6de57fcbf6d9462c518ffeba1d4e4ff29446c3360707dc032\": container with ID starting with 5e44b6213eddedd6de57fcbf6d9462c518ffeba1d4e4ff29446c3360707dc032 not found: ID does not exist" containerID="5e44b6213eddedd6de57fcbf6d9462c518ffeba1d4e4ff29446c3360707dc032"
Aug 04 03:58:19.646141 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.646081    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e44b6213eddedd6de57fcbf6d9462c518ffeba1d4e4ff29446c3360707dc032"} err="failed to get container status \"5e44b6213eddedd6de57fcbf6d9462c518ffeba1d4e4ff29446c3360707dc032\": rpc error: code = NotFound desc = could not find container \"5e44b6213eddedd6de57fcbf6d9462c518ffeba1d4e4ff29446c3360707dc032\": container with ID starting with 5e44b6213eddedd6de57fcbf6d9462c518ffeba1d4e4ff29446c3360707dc032 not found: ID does not exist"
Aug 04 03:58:19.646141 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.646106    2579 scope.go:117] "RemoveContainer" containerID="db8032979b43156a4ceb0b6969b5f7bec89e4e43d28d888c3cde7a7963c69888"
Aug 04 03:58:19.657727 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.657706    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc"]
Aug 04 03:58:19.663921 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.663901    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a60de-74f7cc785b-lprhc"]
Aug 04 03:58:19.676756 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.676724    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"]
Aug 04 03:58:19.678837 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.678819    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a60de-75d547c6d5-rs4fm"]
Aug 04 03:58:19.845072 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.844997    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d1cfa30-e11d-4827-9976-80421e61ea7e" path="/var/lib/kubelet/pods/2d1cfa30-e11d-4827-9976-80421e61ea7e/volumes"
Aug 04 03:58:19.845383 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:19.845370    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f99bf947-b980-413a-82d5-a828eca96122" path="/var/lib/kubelet/pods/f99bf947-b980-413a-82d5-a828eca96122/volumes"
Aug 04 03:58:35.921878 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:35.921845    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"]
Aug 04 03:58:35.922403 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:35.922366    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2" podUID="66ec454e-e5fc-478d-9964-0d01253ab42a" containerName="maas-api" containerID="cri-o://4a2921c0d28980feff099b95660bb5c19e1fd9964e5dfd5891870996c0d10891" gracePeriod=30
Aug 04 03:58:35.953653 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:35.953610    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2" podUID="66ec454e-e5fc-478d-9964-0d01253ab42a" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.59:8443/health\": dial tcp 10.132.0.59:8443: connect: connection refused"
Aug 04 03:58:35.973440 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:35.973407    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j"]
Aug 04 03:58:35.973665 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:35.973624    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j" podUID="ea58dc82-7669-4a88-94cf-64a4898c61a6" containerName="payload-processing" containerID="cri-o://6755112caec4e8fa84f2d5198ea2d3e4b6e1d0a9edff9b58b304649187a87f8e" gracePeriod=30
Aug 04 03:58:35.985180 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:35.985140    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h"]
Aug 04 03:58:35.985598 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:35.985524    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h" podUID="f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad" containerName="payload-pre-processing" containerID="cri-o://8bd2c0ba922bdcbd4ab4d3cf7d36c1179aa8b17a80afce103a68556a1f3f4e0d" gracePeriod=30
Aug 04 03:58:36.273049 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.273019    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j"
Aug 04 03:58:36.277169 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.277140    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h"
Aug 04 03:58:36.377681 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.377648    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw9ff\" (UniqueName: \"kubernetes.io/projected/f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad-kube-api-access-bw9ff\") pod \"f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad\" (UID: \"f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad\") "
Aug 04 03:58:36.377681 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.377686    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea58dc82-7669-4a88-94cf-64a4898c61a6-plugins-config-volume\") pod \"ea58dc82-7669-4a88-94cf-64a4898c61a6\" (UID: \"ea58dc82-7669-4a88-94cf-64a4898c61a6\") "
Aug 04 03:58:36.377923 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.377777    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad-plugins-config-volume\") pod \"f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad\" (UID: \"f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad\") "
Aug 04 03:58:36.378002 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.377948    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7h92\" (UniqueName: \"kubernetes.io/projected/ea58dc82-7669-4a88-94cf-64a4898c61a6-kube-api-access-c7h92\") pod \"ea58dc82-7669-4a88-94cf-64a4898c61a6\" (UID: \"ea58dc82-7669-4a88-94cf-64a4898c61a6\") "
Aug 04 03:58:36.378151 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.378125    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad" (UID: "f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:58:36.378277 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.378160    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea58dc82-7669-4a88-94cf-64a4898c61a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea58dc82-7669-4a88-94cf-64a4898c61a6" (UID: "ea58dc82-7669-4a88-94cf-64a4898c61a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:58:36.379359 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.379337    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:58:36.379511 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.379497    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea58dc82-7669-4a88-94cf-64a4898c61a6-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:58:36.381231 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.381207    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea58dc82-7669-4a88-94cf-64a4898c61a6-kube-api-access-c7h92" (OuterVolumeSpecName: "kube-api-access-c7h92") pod "ea58dc82-7669-4a88-94cf-64a4898c61a6" (UID: "ea58dc82-7669-4a88-94cf-64a4898c61a6"). InnerVolumeSpecName "kube-api-access-c7h92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:58:36.383002 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.381399    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad-kube-api-access-bw9ff" (OuterVolumeSpecName: "kube-api-access-bw9ff") pod "f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad" (UID: "f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad"). InnerVolumeSpecName "kube-api-access-bw9ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:58:36.403049 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.403020    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"
Aug 04 03:58:36.480646 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.480607    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ec454e-e5fc-478d-9964-0d01253ab42a-openshift-service-ca\") pod \"66ec454e-e5fc-478d-9964-0d01253ab42a\" (UID: \"66ec454e-e5fc-478d-9964-0d01253ab42a\") "
Aug 04 03:58:36.480819 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.480663    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmg7l\" (UniqueName: \"kubernetes.io/projected/66ec454e-e5fc-478d-9964-0d01253ab42a-kube-api-access-mmg7l\") pod \"66ec454e-e5fc-478d-9964-0d01253ab42a\" (UID: \"66ec454e-e5fc-478d-9964-0d01253ab42a\") "
Aug 04 03:58:36.480819 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.480702    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66ec454e-e5fc-478d-9964-0d01253ab42a-maas-api-tls\") pod \"66ec454e-e5fc-478d-9964-0d01253ab42a\" (UID: \"66ec454e-e5fc-478d-9964-0d01253ab42a\") "
Aug 04 03:58:36.480944 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.480889    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7h92\" (UniqueName: \"kubernetes.io/projected/ea58dc82-7669-4a88-94cf-64a4898c61a6-kube-api-access-c7h92\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:58:36.480944 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.480900    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw9ff\" (UniqueName: \"kubernetes.io/projected/f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad-kube-api-access-bw9ff\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:58:36.481097 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.480955    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ec454e-e5fc-478d-9964-0d01253ab42a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66ec454e-e5fc-478d-9964-0d01253ab42a" (UID: "66ec454e-e5fc-478d-9964-0d01253ab42a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:58:36.482804 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.482781    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ec454e-e5fc-478d-9964-0d01253ab42a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66ec454e-e5fc-478d-9964-0d01253ab42a" (UID: "66ec454e-e5fc-478d-9964-0d01253ab42a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:58:36.482887 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.482782    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ec454e-e5fc-478d-9964-0d01253ab42a-kube-api-access-mmg7l" (OuterVolumeSpecName: "kube-api-access-mmg7l") pod "66ec454e-e5fc-478d-9964-0d01253ab42a" (UID: "66ec454e-e5fc-478d-9964-0d01253ab42a"). InnerVolumeSpecName "kube-api-access-mmg7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:58:36.581829 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.581792    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmg7l\" (UniqueName: \"kubernetes.io/projected/66ec454e-e5fc-478d-9964-0d01253ab42a-kube-api-access-mmg7l\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:58:36.581829 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.581821    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66ec454e-e5fc-478d-9964-0d01253ab42a-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:58:36.581829 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.581830    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ec454e-e5fc-478d-9964-0d01253ab42a-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:58:36.724728 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.724696    2579 generic.go:356] "Generic (PLEG): container finished" podID="f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad" containerID="8bd2c0ba922bdcbd4ab4d3cf7d36c1179aa8b17a80afce103a68556a1f3f4e0d" exitCode=0
Aug 04 03:58:36.724982 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.724755    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h"
Aug 04 03:58:36.724982 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.724776    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h" event={"ID":"f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad","Type":"ContainerDied","Data":"8bd2c0ba922bdcbd4ab4d3cf7d36c1179aa8b17a80afce103a68556a1f3f4e0d"}
Aug 04 03:58:36.724982 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.724854    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h" event={"ID":"f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad","Type":"ContainerDied","Data":"555e3ce7a4f2fd91813700edb7ea12e0258aac7cac7839d9a2b22961958c8b8d"}
Aug 04 03:58:36.724982 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.724884    2579 scope.go:117] "RemoveContainer" containerID="8bd2c0ba922bdcbd4ab4d3cf7d36c1179aa8b17a80afce103a68556a1f3f4e0d"
Aug 04 03:58:36.726200 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.726171    2579 generic.go:356] "Generic (PLEG): container finished" podID="ea58dc82-7669-4a88-94cf-64a4898c61a6" containerID="6755112caec4e8fa84f2d5198ea2d3e4b6e1d0a9edff9b58b304649187a87f8e" exitCode=0
Aug 04 03:58:36.726334 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.726225    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j"
Aug 04 03:58:36.726334 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.726253    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j" event={"ID":"ea58dc82-7669-4a88-94cf-64a4898c61a6","Type":"ContainerDied","Data":"6755112caec4e8fa84f2d5198ea2d3e4b6e1d0a9edff9b58b304649187a87f8e"}
Aug 04 03:58:36.726334 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.726312    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j" event={"ID":"ea58dc82-7669-4a88-94cf-64a4898c61a6","Type":"ContainerDied","Data":"8d8212e0b1167900f6cc378fac19ee2162e75c79a67410774e11fbb7e57318b0"}
Aug 04 03:58:36.727611 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.727586    2579 generic.go:356] "Generic (PLEG): container finished" podID="66ec454e-e5fc-478d-9964-0d01253ab42a" containerID="4a2921c0d28980feff099b95660bb5c19e1fd9964e5dfd5891870996c0d10891" exitCode=0
Aug 04 03:58:36.727709 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.727618    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"
Aug 04 03:58:36.727709 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.727628    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2" event={"ID":"66ec454e-e5fc-478d-9964-0d01253ab42a","Type":"ContainerDied","Data":"4a2921c0d28980feff099b95660bb5c19e1fd9964e5dfd5891870996c0d10891"}
Aug 04 03:58:36.727829 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.727711    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2" event={"ID":"66ec454e-e5fc-478d-9964-0d01253ab42a","Type":"ContainerDied","Data":"c6941f807a2ba7a4eff23e74f09911ee3a30bd412439be1ba1cb55c041952830"}
Aug 04 03:58:36.735481 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.735442    2579 scope.go:117] "RemoveContainer" containerID="8bd2c0ba922bdcbd4ab4d3cf7d36c1179aa8b17a80afce103a68556a1f3f4e0d"
Aug 04 03:58:36.735766 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:58:36.735743    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd2c0ba922bdcbd4ab4d3cf7d36c1179aa8b17a80afce103a68556a1f3f4e0d\": container with ID starting with 8bd2c0ba922bdcbd4ab4d3cf7d36c1179aa8b17a80afce103a68556a1f3f4e0d not found: ID does not exist" containerID="8bd2c0ba922bdcbd4ab4d3cf7d36c1179aa8b17a80afce103a68556a1f3f4e0d"
Aug 04 03:58:36.735827 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.735779    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd2c0ba922bdcbd4ab4d3cf7d36c1179aa8b17a80afce103a68556a1f3f4e0d"} err="failed to get container status \"8bd2c0ba922bdcbd4ab4d3cf7d36c1179aa8b17a80afce103a68556a1f3f4e0d\": rpc error: code = NotFound desc = could not find container \"8bd2c0ba922bdcbd4ab4d3cf7d36c1179aa8b17a80afce103a68556a1f3f4e0d\": container with ID starting with 8bd2c0ba922bdcbd4ab4d3cf7d36c1179aa8b17a80afce103a68556a1f3f4e0d not found: ID does not exist"
Aug 04 03:58:36.735827 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.735804    2579 scope.go:117] "RemoveContainer" containerID="6755112caec4e8fa84f2d5198ea2d3e4b6e1d0a9edff9b58b304649187a87f8e"
Aug 04 03:58:36.744056 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.744040    2579 scope.go:117] "RemoveContainer" containerID="6755112caec4e8fa84f2d5198ea2d3e4b6e1d0a9edff9b58b304649187a87f8e"
Aug 04 03:58:36.744336 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:58:36.744314    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6755112caec4e8fa84f2d5198ea2d3e4b6e1d0a9edff9b58b304649187a87f8e\": container with ID starting with 6755112caec4e8fa84f2d5198ea2d3e4b6e1d0a9edff9b58b304649187a87f8e not found: ID does not exist" containerID="6755112caec4e8fa84f2d5198ea2d3e4b6e1d0a9edff9b58b304649187a87f8e"
Aug 04 03:58:36.744394 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.744345    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6755112caec4e8fa84f2d5198ea2d3e4b6e1d0a9edff9b58b304649187a87f8e"} err="failed to get container status \"6755112caec4e8fa84f2d5198ea2d3e4b6e1d0a9edff9b58b304649187a87f8e\": rpc error: code = NotFound desc = could not find container \"6755112caec4e8fa84f2d5198ea2d3e4b6e1d0a9edff9b58b304649187a87f8e\": container with ID starting with 6755112caec4e8fa84f2d5198ea2d3e4b6e1d0a9edff9b58b304649187a87f8e not found: ID does not exist"
Aug 04 03:58:36.744394 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.744366    2579 scope.go:117] "RemoveContainer" containerID="4a2921c0d28980feff099b95660bb5c19e1fd9964e5dfd5891870996c0d10891"
Aug 04 03:58:36.752026 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.752009    2579 scope.go:117] "RemoveContainer" containerID="4a2921c0d28980feff099b95660bb5c19e1fd9964e5dfd5891870996c0d10891"
Aug 04 03:58:36.752279 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:58:36.752260    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a2921c0d28980feff099b95660bb5c19e1fd9964e5dfd5891870996c0d10891\": container with ID starting with 4a2921c0d28980feff099b95660bb5c19e1fd9964e5dfd5891870996c0d10891 not found: ID does not exist" containerID="4a2921c0d28980feff099b95660bb5c19e1fd9964e5dfd5891870996c0d10891"
Aug 04 03:58:36.752326 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.752285    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a2921c0d28980feff099b95660bb5c19e1fd9964e5dfd5891870996c0d10891"} err="failed to get container status \"4a2921c0d28980feff099b95660bb5c19e1fd9964e5dfd5891870996c0d10891\": rpc error: code = NotFound desc = could not find container \"4a2921c0d28980feff099b95660bb5c19e1fd9964e5dfd5891870996c0d10891\": container with ID starting with 4a2921c0d28980feff099b95660bb5c19e1fd9964e5dfd5891870996c0d10891 not found: ID does not exist"
Aug 04 03:58:36.760708 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.760685    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j"]
Aug 04 03:58:36.762731 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.762711    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e38897-cd54968cb-tsn9j"]
Aug 04 03:58:36.774858 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.774835    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h"]
Aug 04 03:58:36.777277 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.777247    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e38897-7db7f69788-2l44h"]
Aug 04 03:58:36.787522 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.787500    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"]
Aug 04 03:58:36.793435 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:36.793409    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e38897-57b78998dc-rvmk2"]
Aug 04 03:58:37.845177 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:37.845147    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66ec454e-e5fc-478d-9964-0d01253ab42a" path="/var/lib/kubelet/pods/66ec454e-e5fc-478d-9964-0d01253ab42a/volumes"
Aug 04 03:58:37.845560 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:37.845514    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea58dc82-7669-4a88-94cf-64a4898c61a6" path="/var/lib/kubelet/pods/ea58dc82-7669-4a88-94cf-64a4898c61a6/volumes"
Aug 04 03:58:37.845831 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:37.845818    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad" path="/var/lib/kubelet/pods/f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad/volumes"
Aug 04 03:58:47.705864 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.705832    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"]
Aug 04 03:58:47.706246 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.706237    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad" containerName="payload-pre-processing"
Aug 04 03:58:47.706285 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.706250    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad" containerName="payload-pre-processing"
Aug 04 03:58:47.706285 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.706268    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f99bf947-b980-413a-82d5-a828eca96122" containerName="payload-processing"
Aug 04 03:58:47.706285 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.706274    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f99bf947-b980-413a-82d5-a828eca96122" containerName="payload-processing"
Aug 04 03:58:47.706285 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.706284    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ec454e-e5fc-478d-9964-0d01253ab42a" containerName="maas-api"
Aug 04 03:58:47.706430 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.706290    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ec454e-e5fc-478d-9964-0d01253ab42a" containerName="maas-api"
Aug 04 03:58:47.706430 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.706308    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea58dc82-7669-4a88-94cf-64a4898c61a6" containerName="payload-processing"
Aug 04 03:58:47.706430 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.706314    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea58dc82-7669-4a88-94cf-64a4898c61a6" containerName="payload-processing"
Aug 04 03:58:47.706430 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.706325    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1cfa30-e11d-4827-9976-80421e61ea7e" containerName="maas-api"
Aug 04 03:58:47.706430 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.706331    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1cfa30-e11d-4827-9976-80421e61ea7e" containerName="maas-api"
Aug 04 03:58:47.706430 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.706385    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f99bf947-b980-413a-82d5-a828eca96122" containerName="payload-processing"
Aug 04 03:58:47.706430 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.706396    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ec454e-e5fc-478d-9964-0d01253ab42a" containerName="maas-api"
Aug 04 03:58:47.706430 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.706402    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4e1d7e2-b43d-4fe4-bc6c-5a857a2a5dad" containerName="payload-pre-processing"
Aug 04 03:58:47.706430 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.706409    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1cfa30-e11d-4827-9976-80421e61ea7e" containerName="maas-api"
Aug 04 03:58:47.706430 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.706416    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea58dc82-7669-4a88-94cf-64a4898c61a6" containerName="payload-processing"
Aug 04 03:58:47.709378 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.709360    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"
Aug 04 03:58:47.711689 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.711668    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ccfdb2-serving-cert\""
Aug 04 03:58:47.718232 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.718211    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"]
Aug 04 03:58:47.787414 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.787379    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62fe8c2-0569-4054-8142-45784c78a8aa-maas-api-tls\") pod \"maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8\" (UID: \"d62fe8c2-0569-4054-8142-45784c78a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"
Aug 04 03:58:47.787624 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.787555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgl28\" (UniqueName: \"kubernetes.io/projected/d62fe8c2-0569-4054-8142-45784c78a8aa-kube-api-access-fgl28\") pod \"maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8\" (UID: \"d62fe8c2-0569-4054-8142-45784c78a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"
Aug 04 03:58:47.787624 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.787593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62fe8c2-0569-4054-8142-45784c78a8aa-openshift-service-ca\") pod \"maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8\" (UID: \"d62fe8c2-0569-4054-8142-45784c78a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"
Aug 04 03:58:47.888199 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.888164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62fe8c2-0569-4054-8142-45784c78a8aa-maas-api-tls\") pod \"maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8\" (UID: \"d62fe8c2-0569-4054-8142-45784c78a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"
Aug 04 03:58:47.888378 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.888263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgl28\" (UniqueName: \"kubernetes.io/projected/d62fe8c2-0569-4054-8142-45784c78a8aa-kube-api-access-fgl28\") pod \"maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8\" (UID: \"d62fe8c2-0569-4054-8142-45784c78a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"
Aug 04 03:58:47.888378 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.888286    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62fe8c2-0569-4054-8142-45784c78a8aa-openshift-service-ca\") pod \"maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8\" (UID: \"d62fe8c2-0569-4054-8142-45784c78a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"
Aug 04 03:58:47.888378 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:58:47.888306    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-serving-cert: secret "maas-api-e2e-api-a-ccfdb2-serving-cert" not found
Aug 04 03:58:47.888378 ip-10-0-138-99 kubenswrapper[2579]: E0804 03:58:47.888369    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d62fe8c2-0569-4054-8142-45784c78a8aa-maas-api-tls podName:d62fe8c2-0569-4054-8142-45784c78a8aa nodeName:}" failed. No retries permitted until 2026-08-04 03:58:48.388353914 +0000 UTC m=+2959.068141011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d62fe8c2-0569-4054-8142-45784c78a8aa-maas-api-tls") pod "maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8" (UID: "d62fe8c2-0569-4054-8142-45784c78a8aa") : secret "maas-api-e2e-api-a-ccfdb2-serving-cert" not found
Aug 04 03:58:47.888984 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.888951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62fe8c2-0569-4054-8142-45784c78a8aa-openshift-service-ca\") pod \"maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8\" (UID: \"d62fe8c2-0569-4054-8142-45784c78a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"
Aug 04 03:58:47.896654 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:47.896635    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgl28\" (UniqueName: \"kubernetes.io/projected/d62fe8c2-0569-4054-8142-45784c78a8aa-kube-api-access-fgl28\") pod \"maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8\" (UID: \"d62fe8c2-0569-4054-8142-45784c78a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"
Aug 04 03:58:48.393714 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:48.393682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62fe8c2-0569-4054-8142-45784c78a8aa-maas-api-tls\") pod \"maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8\" (UID: \"d62fe8c2-0569-4054-8142-45784c78a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"
Aug 04 03:58:48.394875 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:48.394851    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z"]
Aug 04 03:58:48.396314 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:48.396292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62fe8c2-0569-4054-8142-45784c78a8aa-maas-api-tls\") pod \"maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8\" (UID: \"d62fe8c2-0569-4054-8142-45784c78a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"
Aug 04 03:58:48.399308 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:48.399291    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z"
Aug 04 03:58:48.401943 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:48.401920    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ccfdb2\""
Aug 04 03:58:48.402052 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:48.401941    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ccfdb2-dockercfg-6pq9b\""
Aug 04 03:58:48.408113 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:48.408092    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z"]
Aug 04 03:58:48.494795 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:48.494759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ac9d0f-3ef8-4690-964f-d7a1fe553642-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z\" (UID: \"32ac9d0f-3ef8-4690-964f-d7a1fe553642\") " pod="openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z"
Aug 04 03:58:48.495006 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:48.494816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc5rg\" (UniqueName: \"kubernetes.io/projected/32ac9d0f-3ef8-4690-964f-d7a1fe553642-kube-api-access-xc5rg\") pod \"payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z\" (UID: \"32ac9d0f-3ef8-4690-964f-d7a1fe553642\") " pod="openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z"
Aug 04 03:58:48.596028 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:48.595988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ac9d0f-3ef8-4690-964f-d7a1fe553642-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z\" (UID: \"32ac9d0f-3ef8-4690-964f-d7a1fe553642\") " pod="openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z"
Aug 04 03:58:48.596271 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:48.596151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc5rg\" (UniqueName: \"kubernetes.io/projected/32ac9d0f-3ef8-4690-964f-d7a1fe553642-kube-api-access-xc5rg\") pod \"payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z\" (UID: \"32ac9d0f-3ef8-4690-964f-d7a1fe553642\") " pod="openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z"
Aug 04 03:58:48.596798 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:48.596774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ac9d0f-3ef8-4690-964f-d7a1fe553642-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z\" (UID: \"32ac9d0f-3ef8-4690-964f-d7a1fe553642\") " pod="openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z"
Aug 04 03:58:48.604037 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:48.604013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc5rg\" (UniqueName: \"kubernetes.io/projected/32ac9d0f-3ef8-4690-964f-d7a1fe553642-kube-api-access-xc5rg\") pod \"payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z\" (UID: \"32ac9d0f-3ef8-4690-964f-d7a1fe553642\") " pod="openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z"
Aug 04 03:58:48.620864 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:48.620843    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"
Aug 04 03:58:48.716286 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:48.716241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z"
Aug 04 03:58:48.766168 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:48.766042    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"]
Aug 04 03:58:48.768511 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:58:48.768487    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62fe8c2_0569_4054_8142_45784c78a8aa.slice/crio-cc63439da75c8cc6d62f69f6a5293da08612a83204ffa786d570014661fd93d2 WatchSource:0}: Error finding container cc63439da75c8cc6d62f69f6a5293da08612a83204ffa786d570014661fd93d2: Status 404 returned error can't find the container with id cc63439da75c8cc6d62f69f6a5293da08612a83204ffa786d570014661fd93d2
Aug 04 03:58:48.770042 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:48.770022    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 03:58:48.799410 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:48.799376    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8" event={"ID":"d62fe8c2-0569-4054-8142-45784c78a8aa","Type":"ContainerStarted","Data":"cc63439da75c8cc6d62f69f6a5293da08612a83204ffa786d570014661fd93d2"}
Aug 04 03:58:48.861409 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:48.861378    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z"]
Aug 04 03:58:48.864827 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:58:48.864799    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ac9d0f_3ef8_4690_964f_d7a1fe553642.slice/crio-f433918ed37671b61c2a89a814c2d55e745bff4dfb50eddf722e7ea5cbc3bfe9 WatchSource:0}: Error finding container f433918ed37671b61c2a89a814c2d55e745bff4dfb50eddf722e7ea5cbc3bfe9: Status 404 returned error can't find the container with id f433918ed37671b61c2a89a814c2d55e745bff4dfb50eddf722e7ea5cbc3bfe9
Aug 04 03:58:49.807036 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:49.807001    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8" event={"ID":"d62fe8c2-0569-4054-8142-45784c78a8aa","Type":"ContainerStarted","Data":"898d31562c70e319a895831a456bf853d6510804e233a8d4db4ad4d45b56c34e"}
Aug 04 03:58:49.807546 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:49.807098    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"
Aug 04 03:58:49.808514 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:49.808491    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z" event={"ID":"32ac9d0f-3ef8-4690-964f-d7a1fe553642","Type":"ContainerStarted","Data":"fcb1bb670bbc72e5de960b3384e07d06620d0b551f26c5b22d8e9c330ee1257b"}
Aug 04 03:58:49.808644 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:49.808528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z" event={"ID":"32ac9d0f-3ef8-4690-964f-d7a1fe553642","Type":"ContainerStarted","Data":"f433918ed37671b61c2a89a814c2d55e745bff4dfb50eddf722e7ea5cbc3bfe9"}
Aug 04 03:58:49.808644 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:49.808564    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z"
Aug 04 03:58:49.824975 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:49.824919    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8" podStartSLOduration=2.392050363 podStartE2EDuration="2.824903934s" totalImagesPullingTime="432.853571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:58:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:58:49.823516206 +0000 UTC m=+2960.503303347" watchObservedRunningTime="2026-08-04 03:58:49.824903934 +0000 UTC m=+2960.504691052"
Aug 04 03:58:49.841172 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:49.841126    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z" podStartSLOduration=1.841112827 podStartE2EDuration="1.841112827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:58:49.838918864 +0000 UTC m=+2960.518705984" watchObservedRunningTime="2026-08-04 03:58:49.841112827 +0000 UTC m=+2960.520899947"
Aug 04 03:58:55.820838 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:58:55.820751    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"
Aug 04 03:59:00.817908 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:00.817875    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z"
Aug 04 03:59:16.882481 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:16.882446    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"]
Aug 04 03:59:16.886292 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:16.886276    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"
Aug 04 03:59:16.888521 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:16.888502    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-523226-serving-cert\""
Aug 04 03:59:16.895190 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:16.895166    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"]
Aug 04 03:59:16.955728 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:16.955693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xs7l\" (UniqueName: \"kubernetes.io/projected/7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca-kube-api-access-8xs7l\") pod \"maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px\" (UID: \"7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"
Aug 04 03:59:16.955905 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:16.955770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca-openshift-service-ca\") pod \"maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px\" (UID: \"7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"
Aug 04 03:59:16.955980 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:16.955908    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca-maas-api-tls\") pod \"maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px\" (UID: \"7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"
Aug 04 03:59:17.056442 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.056401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca-openshift-service-ca\") pod \"maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px\" (UID: \"7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"
Aug 04 03:59:17.056631 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.056519    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca-maas-api-tls\") pod \"maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px\" (UID: \"7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"
Aug 04 03:59:17.056631 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.056578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xs7l\" (UniqueName: \"kubernetes.io/projected/7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca-kube-api-access-8xs7l\") pod \"maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px\" (UID: \"7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"
Aug 04 03:59:17.057073 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.057051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca-openshift-service-ca\") pod \"maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px\" (UID: \"7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"
Aug 04 03:59:17.058908 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.058886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca-maas-api-tls\") pod \"maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px\" (UID: \"7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"
Aug 04 03:59:17.072178 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.072155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xs7l\" (UniqueName: \"kubernetes.io/projected/7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca-kube-api-access-8xs7l\") pod \"maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px\" (UID: \"7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"
Aug 04 03:59:17.198329 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.198219    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"
Aug 04 03:59:17.367424 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.367377    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"]
Aug 04 03:59:17.369690 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:59:17.369660    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbd272a_2aeb_413d_89d5_76f8b0d9d6ca.slice/crio-14dd75ac92db0d10510128f4804eb6eae0fd4bd406e8828535a811efbcc03741 WatchSource:0}: Error finding container 14dd75ac92db0d10510128f4804eb6eae0fd4bd406e8828535a811efbcc03741: Status 404 returned error can't find the container with id 14dd75ac92db0d10510128f4804eb6eae0fd4bd406e8828535a811efbcc03741
Aug 04 03:59:17.552116 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.552077    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86"]
Aug 04 03:59:17.556772 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.556755    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86"
Aug 04 03:59:17.559820 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.559797    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-523226\""
Aug 04 03:59:17.559989 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.559858    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-523226-dockercfg-tgfnk\""
Aug 04 03:59:17.566454 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.566387    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86"]
Aug 04 03:59:17.578279 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.578250    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw"]
Aug 04 03:59:17.582477 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.582453    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw"
Aug 04 03:59:17.591540 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.591517    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw"]
Aug 04 03:59:17.664874 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.664838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x2ft\" (UniqueName: \"kubernetes.io/projected/d0bc8f75-bb42-4be3-b3bc-456ef2318980-kube-api-access-6x2ft\") pod \"payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw\" (UID: \"d0bc8f75-bb42-4be3-b3bc-456ef2318980\") " pod="openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw"
Aug 04 03:59:17.665328 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.664902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/364e00c6-b98b-47d9-9719-3e563a2ab487-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86\" (UID: \"364e00c6-b98b-47d9-9719-3e563a2ab487\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86"
Aug 04 03:59:17.665328 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.665038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0bc8f75-bb42-4be3-b3bc-456ef2318980-plugins-config-volume\") pod \"payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw\" (UID: \"d0bc8f75-bb42-4be3-b3bc-456ef2318980\") " pod="openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw"
Aug 04 03:59:17.665328 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.665099    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf6g5\" (UniqueName: \"kubernetes.io/projected/364e00c6-b98b-47d9-9719-3e563a2ab487-kube-api-access-cf6g5\") pod \"payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86\" (UID: \"364e00c6-b98b-47d9-9719-3e563a2ab487\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86"
Aug 04 03:59:17.766412 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.766368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0bc8f75-bb42-4be3-b3bc-456ef2318980-plugins-config-volume\") pod \"payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw\" (UID: \"d0bc8f75-bb42-4be3-b3bc-456ef2318980\") " pod="openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw"
Aug 04 03:59:17.766626 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.766447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf6g5\" (UniqueName: \"kubernetes.io/projected/364e00c6-b98b-47d9-9719-3e563a2ab487-kube-api-access-cf6g5\") pod \"payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86\" (UID: \"364e00c6-b98b-47d9-9719-3e563a2ab487\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86"
Aug 04 03:59:17.766626 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.766494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x2ft\" (UniqueName: \"kubernetes.io/projected/d0bc8f75-bb42-4be3-b3bc-456ef2318980-kube-api-access-6x2ft\") pod \"payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw\" (UID: \"d0bc8f75-bb42-4be3-b3bc-456ef2318980\") " pod="openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw"
Aug 04 03:59:17.766763 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.766682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/364e00c6-b98b-47d9-9719-3e563a2ab487-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86\" (UID: \"364e00c6-b98b-47d9-9719-3e563a2ab487\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86"
Aug 04 03:59:17.767234 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.767215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0bc8f75-bb42-4be3-b3bc-456ef2318980-plugins-config-volume\") pod \"payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw\" (UID: \"d0bc8f75-bb42-4be3-b3bc-456ef2318980\") " pod="openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw"
Aug 04 03:59:17.767344 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.767326    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/364e00c6-b98b-47d9-9719-3e563a2ab487-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86\" (UID: \"364e00c6-b98b-47d9-9719-3e563a2ab487\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86"
Aug 04 03:59:17.774826 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.774800    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf6g5\" (UniqueName: \"kubernetes.io/projected/364e00c6-b98b-47d9-9719-3e563a2ab487-kube-api-access-cf6g5\") pod \"payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86\" (UID: \"364e00c6-b98b-47d9-9719-3e563a2ab487\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86"
Aug 04 03:59:17.774927 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.774840    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x2ft\" (UniqueName: \"kubernetes.io/projected/d0bc8f75-bb42-4be3-b3bc-456ef2318980-kube-api-access-6x2ft\") pod \"payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw\" (UID: \"d0bc8f75-bb42-4be3-b3bc-456ef2318980\") " pod="openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw"
Aug 04 03:59:17.869722 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.869688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86"
Aug 04 03:59:17.894807 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.894771    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw"
Aug 04 03:59:17.962673 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.962418    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px" event={"ID":"7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca","Type":"ContainerStarted","Data":"67da5cdf8b7a4417bfcc5d7d1d0b90f285f3534b87844dd34761e40b0807165d"}
Aug 04 03:59:17.962673 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.962537    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px" event={"ID":"7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca","Type":"ContainerStarted","Data":"14dd75ac92db0d10510128f4804eb6eae0fd4bd406e8828535a811efbcc03741"}
Aug 04 03:59:17.962923 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.962786    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"
Aug 04 03:59:17.983247 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:17.983157    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px" podStartSLOduration=1.5194945180000001 podStartE2EDuration="1.983134698s" totalImagesPullingTime="463.64018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:59:17.981180754 +0000 UTC m=+2988.660967901" watchObservedRunningTime="2026-08-04 03:59:17.983134698 +0000 UTC m=+2988.662921820"
Aug 04 03:59:18.037864 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:18.037271    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86"]
Aug 04 03:59:18.041067 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:59:18.041023    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364e00c6_b98b_47d9_9719_3e563a2ab487.slice/crio-046421cd29af1e57a8d867a2c0985f76c3ce3a62b38513608330458d6eb8cf7c WatchSource:0}: Error finding container 046421cd29af1e57a8d867a2c0985f76c3ce3a62b38513608330458d6eb8cf7c: Status 404 returned error can't find the container with id 046421cd29af1e57a8d867a2c0985f76c3ce3a62b38513608330458d6eb8cf7c
Aug 04 03:59:18.067648 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:18.067623    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw"]
Aug 04 03:59:18.069503 ip-10-0-138-99 kubenswrapper[2579]: W0804 03:59:18.069479    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0bc8f75_bb42_4be3_b3bc_456ef2318980.slice/crio-a951883240c5b398d27abd3c133724270ee711454acdc0ac68cfed69aff5a53c WatchSource:0}: Error finding container a951883240c5b398d27abd3c133724270ee711454acdc0ac68cfed69aff5a53c: Status 404 returned error can't find the container with id a951883240c5b398d27abd3c133724270ee711454acdc0ac68cfed69aff5a53c
Aug 04 03:59:18.969986 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:18.969935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86" event={"ID":"364e00c6-b98b-47d9-9719-3e563a2ab487","Type":"ContainerStarted","Data":"f66031241a140746de1cf0b347754d9cdb69c7f9abbdcd62ca53526f582851a7"}
Aug 04 03:59:18.970491 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:18.970018    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86" event={"ID":"364e00c6-b98b-47d9-9719-3e563a2ab487","Type":"ContainerStarted","Data":"046421cd29af1e57a8d867a2c0985f76c3ce3a62b38513608330458d6eb8cf7c"}
Aug 04 03:59:18.970491 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:18.970121    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86"
Aug 04 03:59:18.971504 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:18.971480    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw" event={"ID":"d0bc8f75-bb42-4be3-b3bc-456ef2318980","Type":"ContainerStarted","Data":"90b12dd49290b9a6b872a7e55e4f172b494433d87a8194303ff8939d0f437274"}
Aug 04 03:59:18.971602 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:18.971526    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw" event={"ID":"d0bc8f75-bb42-4be3-b3bc-456ef2318980","Type":"ContainerStarted","Data":"a951883240c5b398d27abd3c133724270ee711454acdc0ac68cfed69aff5a53c"}
Aug 04 03:59:18.986906 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:18.986860    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86" podStartSLOduration=1.986844523 podStartE2EDuration="1.986844523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:59:18.985026359 +0000 UTC m=+2989.664813499" watchObservedRunningTime="2026-08-04 03:59:18.986844523 +0000 UTC m=+2989.666631643"
Aug 04 03:59:19.001363 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:19.001315    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw" podStartSLOduration=2.001297647 podStartE2EDuration="2.001297647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:59:18.999640795 +0000 UTC m=+2989.679427961" watchObservedRunningTime="2026-08-04 03:59:19.001297647 +0000 UTC m=+2989.681084766"
Aug 04 03:59:19.976657 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:19.976615    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw"
Aug 04 03:59:23.977038 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:23.977007    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"
Aug 04 03:59:29.979119 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:29.979086    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86"
Aug 04 03:59:30.983891 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:30.983859    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw"
Aug 04 03:59:59.278865 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.278833    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"]
Aug 04 03:59:59.279370 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.279065    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8" podUID="d62fe8c2-0569-4054-8142-45784c78a8aa" containerName="maas-api" containerID="cri-o://898d31562c70e319a895831a456bf853d6510804e233a8d4db4ad4d45b56c34e" gracePeriod=30
Aug 04 03:59:59.331360 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.331324    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z"]
Aug 04 03:59:59.331652 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.331592    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z" podUID="32ac9d0f-3ef8-4690-964f-d7a1fe553642" containerName="payload-processing" containerID="cri-o://fcb1bb670bbc72e5de960b3384e07d06620d0b551f26c5b22d8e9c330ee1257b" gracePeriod=30
Aug 04 03:59:59.585612 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.585586    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"
Aug 04 03:59:59.615619 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.615592    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z"
Aug 04 03:59:59.663186 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.663145    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgl28\" (UniqueName: \"kubernetes.io/projected/d62fe8c2-0569-4054-8142-45784c78a8aa-kube-api-access-fgl28\") pod \"d62fe8c2-0569-4054-8142-45784c78a8aa\" (UID: \"d62fe8c2-0569-4054-8142-45784c78a8aa\") "
Aug 04 03:59:59.663384 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.663237    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62fe8c2-0569-4054-8142-45784c78a8aa-maas-api-tls\") pod \"d62fe8c2-0569-4054-8142-45784c78a8aa\" (UID: \"d62fe8c2-0569-4054-8142-45784c78a8aa\") "
Aug 04 03:59:59.663384 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.663275    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ac9d0f-3ef8-4690-964f-d7a1fe553642-plugins-config-volume\") pod \"32ac9d0f-3ef8-4690-964f-d7a1fe553642\" (UID: \"32ac9d0f-3ef8-4690-964f-d7a1fe553642\") "
Aug 04 03:59:59.663384 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.663311    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62fe8c2-0569-4054-8142-45784c78a8aa-openshift-service-ca\") pod \"d62fe8c2-0569-4054-8142-45784c78a8aa\" (UID: \"d62fe8c2-0569-4054-8142-45784c78a8aa\") "
Aug 04 03:59:59.663384 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.663352    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xc5rg\" (UniqueName: \"kubernetes.io/projected/32ac9d0f-3ef8-4690-964f-d7a1fe553642-kube-api-access-xc5rg\") pod \"32ac9d0f-3ef8-4690-964f-d7a1fe553642\" (UID: \"32ac9d0f-3ef8-4690-964f-d7a1fe553642\") "
Aug 04 03:59:59.663687 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.663657    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ac9d0f-3ef8-4690-964f-d7a1fe553642-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "32ac9d0f-3ef8-4690-964f-d7a1fe553642" (UID: "32ac9d0f-3ef8-4690-964f-d7a1fe553642"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:59:59.663740 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.663706    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62fe8c2-0569-4054-8142-45784c78a8aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d62fe8c2-0569-4054-8142-45784c78a8aa" (UID: "d62fe8c2-0569-4054-8142-45784c78a8aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 03:59:59.665417 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.665396    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d62fe8c2-0569-4054-8142-45784c78a8aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d62fe8c2-0569-4054-8142-45784c78a8aa" (UID: "d62fe8c2-0569-4054-8142-45784c78a8aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 03:59:59.665522 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.665450    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ac9d0f-3ef8-4690-964f-d7a1fe553642-kube-api-access-xc5rg" (OuterVolumeSpecName: "kube-api-access-xc5rg") pod "32ac9d0f-3ef8-4690-964f-d7a1fe553642" (UID: "32ac9d0f-3ef8-4690-964f-d7a1fe553642"). InnerVolumeSpecName "kube-api-access-xc5rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:59:59.665522 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.665478    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62fe8c2-0569-4054-8142-45784c78a8aa-kube-api-access-fgl28" (OuterVolumeSpecName: "kube-api-access-fgl28") pod "d62fe8c2-0569-4054-8142-45784c78a8aa" (UID: "d62fe8c2-0569-4054-8142-45784c78a8aa"). InnerVolumeSpecName "kube-api-access-fgl28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 03:59:59.764880 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.764845    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62fe8c2-0569-4054-8142-45784c78a8aa-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:59:59.764880 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.764871    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ac9d0f-3ef8-4690-964f-d7a1fe553642-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:59:59.764880 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.764882    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62fe8c2-0569-4054-8142-45784c78a8aa-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:59:59.765140 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.764894    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc5rg\" (UniqueName: \"kubernetes.io/projected/32ac9d0f-3ef8-4690-964f-d7a1fe553642-kube-api-access-xc5rg\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 03:59:59.765140 ip-10-0-138-99 kubenswrapper[2579]: I0804 03:59:59.764903    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgl28\" (UniqueName: \"kubernetes.io/projected/d62fe8c2-0569-4054-8142-45784c78a8aa-kube-api-access-fgl28\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:00:00.190829 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:00.190743    2579 generic.go:356] "Generic (PLEG): container finished" podID="d62fe8c2-0569-4054-8142-45784c78a8aa" containerID="898d31562c70e319a895831a456bf853d6510804e233a8d4db4ad4d45b56c34e" exitCode=0
Aug 04 04:00:00.190829 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:00.190791    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"
Aug 04 04:00:00.191055 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:00.190830    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8" event={"ID":"d62fe8c2-0569-4054-8142-45784c78a8aa","Type":"ContainerDied","Data":"898d31562c70e319a895831a456bf853d6510804e233a8d4db4ad4d45b56c34e"}
Aug 04 04:00:00.191055 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:00.190883    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8" event={"ID":"d62fe8c2-0569-4054-8142-45784c78a8aa","Type":"ContainerDied","Data":"cc63439da75c8cc6d62f69f6a5293da08612a83204ffa786d570014661fd93d2"}
Aug 04 04:00:00.191055 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:00.190911    2579 scope.go:117] "RemoveContainer" containerID="898d31562c70e319a895831a456bf853d6510804e233a8d4db4ad4d45b56c34e"
Aug 04 04:00:00.192228 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:00.192205    2579 generic.go:356] "Generic (PLEG): container finished" podID="32ac9d0f-3ef8-4690-964f-d7a1fe553642" containerID="fcb1bb670bbc72e5de960b3384e07d06620d0b551f26c5b22d8e9c330ee1257b" exitCode=0
Aug 04 04:00:00.192332 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:00.192261    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z"
Aug 04 04:00:00.192332 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:00.192277    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z" event={"ID":"32ac9d0f-3ef8-4690-964f-d7a1fe553642","Type":"ContainerDied","Data":"fcb1bb670bbc72e5de960b3384e07d06620d0b551f26c5b22d8e9c330ee1257b"}
Aug 04 04:00:00.192450 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:00.192340    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z" event={"ID":"32ac9d0f-3ef8-4690-964f-d7a1fe553642","Type":"ContainerDied","Data":"f433918ed37671b61c2a89a814c2d55e745bff4dfb50eddf722e7ea5cbc3bfe9"}
Aug 04 04:00:00.199951 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:00.199934    2579 scope.go:117] "RemoveContainer" containerID="898d31562c70e319a895831a456bf853d6510804e233a8d4db4ad4d45b56c34e"
Aug 04 04:00:00.200268 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:00:00.200248    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"898d31562c70e319a895831a456bf853d6510804e233a8d4db4ad4d45b56c34e\": container with ID starting with 898d31562c70e319a895831a456bf853d6510804e233a8d4db4ad4d45b56c34e not found: ID does not exist" containerID="898d31562c70e319a895831a456bf853d6510804e233a8d4db4ad4d45b56c34e"
Aug 04 04:00:00.200310 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:00.200276    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"898d31562c70e319a895831a456bf853d6510804e233a8d4db4ad4d45b56c34e"} err="failed to get container status \"898d31562c70e319a895831a456bf853d6510804e233a8d4db4ad4d45b56c34e\": rpc error: code = NotFound desc = could not find container \"898d31562c70e319a895831a456bf853d6510804e233a8d4db4ad4d45b56c34e\": container with ID starting with 898d31562c70e319a895831a456bf853d6510804e233a8d4db4ad4d45b56c34e not found: ID does not exist"
Aug 04 04:00:00.200310 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:00.200293    2579 scope.go:117] "RemoveContainer" containerID="fcb1bb670bbc72e5de960b3384e07d06620d0b551f26c5b22d8e9c330ee1257b"
Aug 04 04:00:00.209353 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:00.209333    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z"]
Aug 04 04:00:00.209418 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:00.209356    2579 scope.go:117] "RemoveContainer" containerID="fcb1bb670bbc72e5de960b3384e07d06620d0b551f26c5b22d8e9c330ee1257b"
Aug 04 04:00:00.209661 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:00:00.209639    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb1bb670bbc72e5de960b3384e07d06620d0b551f26c5b22d8e9c330ee1257b\": container with ID starting with fcb1bb670bbc72e5de960b3384e07d06620d0b551f26c5b22d8e9c330ee1257b not found: ID does not exist" containerID="fcb1bb670bbc72e5de960b3384e07d06620d0b551f26c5b22d8e9c330ee1257b"
Aug 04 04:00:00.209724 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:00.209668    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb1bb670bbc72e5de960b3384e07d06620d0b551f26c5b22d8e9c330ee1257b"} err="failed to get container status \"fcb1bb670bbc72e5de960b3384e07d06620d0b551f26c5b22d8e9c330ee1257b\": rpc error: code = NotFound desc = could not find container \"fcb1bb670bbc72e5de960b3384e07d06620d0b551f26c5b22d8e9c330ee1257b\": container with ID starting with fcb1bb670bbc72e5de960b3384e07d06620d0b551f26c5b22d8e9c330ee1257b not found: ID does not exist"
Aug 04 04:00:00.212420 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:00.212399    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ccfdb2-7bd4c8d4b4-n262z"]
Aug 04 04:00:00.223530 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:00.223507    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"]
Aug 04 04:00:00.226946 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:00.226924    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ccfdb2-77cc49dd77-n6mt8"]
Aug 04 04:00:01.845764 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:01.845730    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32ac9d0f-3ef8-4690-964f-d7a1fe553642" path="/var/lib/kubelet/pods/32ac9d0f-3ef8-4690-964f-d7a1fe553642/volumes"
Aug 04 04:00:01.846194 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:01.846154    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d62fe8c2-0569-4054-8142-45784c78a8aa" path="/var/lib/kubelet/pods/d62fe8c2-0569-4054-8142-45784c78a8aa/volumes"
Aug 04 04:00:21.642433 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:21.642353    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"]
Aug 04 04:00:21.642911 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:21.642560    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px" podUID="7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca" containerName="maas-api" containerID="cri-o://67da5cdf8b7a4417bfcc5d7d1d0b90f285f3534b87844dd34761e40b0807165d" gracePeriod=30
Aug 04 04:00:21.703392 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:21.703362    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw"]
Aug 04 04:00:21.703633 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:21.703583    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw" podUID="d0bc8f75-bb42-4be3-b3bc-456ef2318980" containerName="payload-processing" containerID="cri-o://90b12dd49290b9a6b872a7e55e4f172b494433d87a8194303ff8939d0f437274" gracePeriod=30
Aug 04 04:00:21.713904 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:21.713873    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86"]
Aug 04 04:00:21.714158 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:21.714127    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86" podUID="364e00c6-b98b-47d9-9719-3e563a2ab487" containerName="payload-pre-processing" containerID="cri-o://f66031241a140746de1cf0b347754d9cdb69c7f9abbdcd62ca53526f582851a7" gracePeriod=30
Aug 04 04:00:21.976717 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:21.976678    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"
Aug 04 04:00:22.003597 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.003570    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw"
Aug 04 04:00:22.010931 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.010906    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86"
Aug 04 04:00:22.081539 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.081508    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xs7l\" (UniqueName: \"kubernetes.io/projected/7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca-kube-api-access-8xs7l\") pod \"7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca\" (UID: \"7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca\") "
Aug 04 04:00:22.081539 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.081542    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca-openshift-service-ca\") pod \"7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca\" (UID: \"7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca\") "
Aug 04 04:00:22.081770 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.081595    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca-maas-api-tls\") pod \"7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca\" (UID: \"7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca\") "
Aug 04 04:00:22.081940 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.081913    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca" (UID: "7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 04:00:22.083559 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.083534    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca-kube-api-access-8xs7l" (OuterVolumeSpecName: "kube-api-access-8xs7l") pod "7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca" (UID: "7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca"). InnerVolumeSpecName "kube-api-access-8xs7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:00:22.083645 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.083630    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca" (UID: "7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 04:00:22.182232 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.182142    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0bc8f75-bb42-4be3-b3bc-456ef2318980-plugins-config-volume\") pod \"d0bc8f75-bb42-4be3-b3bc-456ef2318980\" (UID: \"d0bc8f75-bb42-4be3-b3bc-456ef2318980\") "
Aug 04 04:00:22.182232 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.182196    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf6g5\" (UniqueName: \"kubernetes.io/projected/364e00c6-b98b-47d9-9719-3e563a2ab487-kube-api-access-cf6g5\") pod \"364e00c6-b98b-47d9-9719-3e563a2ab487\" (UID: \"364e00c6-b98b-47d9-9719-3e563a2ab487\") "
Aug 04 04:00:22.182433 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.182340    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x2ft\" (UniqueName: \"kubernetes.io/projected/d0bc8f75-bb42-4be3-b3bc-456ef2318980-kube-api-access-6x2ft\") pod \"d0bc8f75-bb42-4be3-b3bc-456ef2318980\" (UID: \"d0bc8f75-bb42-4be3-b3bc-456ef2318980\") "
Aug 04 04:00:22.182433 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.182375    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/364e00c6-b98b-47d9-9719-3e563a2ab487-plugins-config-volume\") pod \"364e00c6-b98b-47d9-9719-3e563a2ab487\" (UID: \"364e00c6-b98b-47d9-9719-3e563a2ab487\") "
Aug 04 04:00:22.182600 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.182520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0bc8f75-bb42-4be3-b3bc-456ef2318980-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0bc8f75-bb42-4be3-b3bc-456ef2318980" (UID: "d0bc8f75-bb42-4be3-b3bc-456ef2318980"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 04:00:22.182769 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.182748    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:00:22.182769 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.182768    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xs7l\" (UniqueName: \"kubernetes.io/projected/7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca-kube-api-access-8xs7l\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:00:22.182949 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.182778    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:00:22.182949 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.182788    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0bc8f75-bb42-4be3-b3bc-456ef2318980-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:00:22.182949 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.182804    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364e00c6-b98b-47d9-9719-3e563a2ab487-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "364e00c6-b98b-47d9-9719-3e563a2ab487" (UID: "364e00c6-b98b-47d9-9719-3e563a2ab487"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 04:00:22.184488 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.184466    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0bc8f75-bb42-4be3-b3bc-456ef2318980-kube-api-access-6x2ft" (OuterVolumeSpecName: "kube-api-access-6x2ft") pod "d0bc8f75-bb42-4be3-b3bc-456ef2318980" (UID: "d0bc8f75-bb42-4be3-b3bc-456ef2318980"). InnerVolumeSpecName "kube-api-access-6x2ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:00:22.184579 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.184471    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364e00c6-b98b-47d9-9719-3e563a2ab487-kube-api-access-cf6g5" (OuterVolumeSpecName: "kube-api-access-cf6g5") pod "364e00c6-b98b-47d9-9719-3e563a2ab487" (UID: "364e00c6-b98b-47d9-9719-3e563a2ab487"). InnerVolumeSpecName "kube-api-access-cf6g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:00:22.283682 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.283644    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x2ft\" (UniqueName: \"kubernetes.io/projected/d0bc8f75-bb42-4be3-b3bc-456ef2318980-kube-api-access-6x2ft\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:00:22.283682 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.283675    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/364e00c6-b98b-47d9-9719-3e563a2ab487-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:00:22.283682 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.283687    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf6g5\" (UniqueName: \"kubernetes.io/projected/364e00c6-b98b-47d9-9719-3e563a2ab487-kube-api-access-cf6g5\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:00:22.310050 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.310012    2579 generic.go:356] "Generic (PLEG): container finished" podID="7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca" containerID="67da5cdf8b7a4417bfcc5d7d1d0b90f285f3534b87844dd34761e40b0807165d" exitCode=0
Aug 04 04:00:22.310236 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.310057    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"
Aug 04 04:00:22.310236 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.310102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px" event={"ID":"7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca","Type":"ContainerDied","Data":"67da5cdf8b7a4417bfcc5d7d1d0b90f285f3534b87844dd34761e40b0807165d"}
Aug 04 04:00:22.310236 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.310149    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px" event={"ID":"7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca","Type":"ContainerDied","Data":"14dd75ac92db0d10510128f4804eb6eae0fd4bd406e8828535a811efbcc03741"}
Aug 04 04:00:22.310236 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.310178    2579 scope.go:117] "RemoveContainer" containerID="67da5cdf8b7a4417bfcc5d7d1d0b90f285f3534b87844dd34761e40b0807165d"
Aug 04 04:00:22.311398 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.311381    2579 generic.go:356] "Generic (PLEG): container finished" podID="364e00c6-b98b-47d9-9719-3e563a2ab487" containerID="f66031241a140746de1cf0b347754d9cdb69c7f9abbdcd62ca53526f582851a7" exitCode=0
Aug 04 04:00:22.311464 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.311423    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86"
Aug 04 04:00:22.311515 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.311481    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86" event={"ID":"364e00c6-b98b-47d9-9719-3e563a2ab487","Type":"ContainerDied","Data":"f66031241a140746de1cf0b347754d9cdb69c7f9abbdcd62ca53526f582851a7"}
Aug 04 04:00:22.311573 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.311521    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86" event={"ID":"364e00c6-b98b-47d9-9719-3e563a2ab487","Type":"ContainerDied","Data":"046421cd29af1e57a8d867a2c0985f76c3ce3a62b38513608330458d6eb8cf7c"}
Aug 04 04:00:22.312934 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.312911    2579 generic.go:356] "Generic (PLEG): container finished" podID="d0bc8f75-bb42-4be3-b3bc-456ef2318980" containerID="90b12dd49290b9a6b872a7e55e4f172b494433d87a8194303ff8939d0f437274" exitCode=0
Aug 04 04:00:22.313196 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.312985    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw"
Aug 04 04:00:22.313196 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.313004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw" event={"ID":"d0bc8f75-bb42-4be3-b3bc-456ef2318980","Type":"ContainerDied","Data":"90b12dd49290b9a6b872a7e55e4f172b494433d87a8194303ff8939d0f437274"}
Aug 04 04:00:22.313196 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.313051    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw" event={"ID":"d0bc8f75-bb42-4be3-b3bc-456ef2318980","Type":"ContainerDied","Data":"a951883240c5b398d27abd3c133724270ee711454acdc0ac68cfed69aff5a53c"}
Aug 04 04:00:22.321250 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.321233    2579 scope.go:117] "RemoveContainer" containerID="67da5cdf8b7a4417bfcc5d7d1d0b90f285f3534b87844dd34761e40b0807165d"
Aug 04 04:00:22.321588 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:00:22.321560    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67da5cdf8b7a4417bfcc5d7d1d0b90f285f3534b87844dd34761e40b0807165d\": container with ID starting with 67da5cdf8b7a4417bfcc5d7d1d0b90f285f3534b87844dd34761e40b0807165d not found: ID does not exist" containerID="67da5cdf8b7a4417bfcc5d7d1d0b90f285f3534b87844dd34761e40b0807165d"
Aug 04 04:00:22.321703 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.321599    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67da5cdf8b7a4417bfcc5d7d1d0b90f285f3534b87844dd34761e40b0807165d"} err="failed to get container status \"67da5cdf8b7a4417bfcc5d7d1d0b90f285f3534b87844dd34761e40b0807165d\": rpc error: code = NotFound desc = could not find container \"67da5cdf8b7a4417bfcc5d7d1d0b90f285f3534b87844dd34761e40b0807165d\": container with ID starting with 67da5cdf8b7a4417bfcc5d7d1d0b90f285f3534b87844dd34761e40b0807165d not found: ID does not exist"
Aug 04 04:00:22.321703 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.321624    2579 scope.go:117] "RemoveContainer" containerID="f66031241a140746de1cf0b347754d9cdb69c7f9abbdcd62ca53526f582851a7"
Aug 04 04:00:22.330319 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.330288    2579 scope.go:117] "RemoveContainer" containerID="f66031241a140746de1cf0b347754d9cdb69c7f9abbdcd62ca53526f582851a7"
Aug 04 04:00:22.330666 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:00:22.330637    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f66031241a140746de1cf0b347754d9cdb69c7f9abbdcd62ca53526f582851a7\": container with ID starting with f66031241a140746de1cf0b347754d9cdb69c7f9abbdcd62ca53526f582851a7 not found: ID does not exist" containerID="f66031241a140746de1cf0b347754d9cdb69c7f9abbdcd62ca53526f582851a7"
Aug 04 04:00:22.330763 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.330675    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66031241a140746de1cf0b347754d9cdb69c7f9abbdcd62ca53526f582851a7"} err="failed to get container status \"f66031241a140746de1cf0b347754d9cdb69c7f9abbdcd62ca53526f582851a7\": rpc error: code = NotFound desc = could not find container \"f66031241a140746de1cf0b347754d9cdb69c7f9abbdcd62ca53526f582851a7\": container with ID starting with f66031241a140746de1cf0b347754d9cdb69c7f9abbdcd62ca53526f582851a7 not found: ID does not exist"
Aug 04 04:00:22.330763 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.330699    2579 scope.go:117] "RemoveContainer" containerID="90b12dd49290b9a6b872a7e55e4f172b494433d87a8194303ff8939d0f437274"
Aug 04 04:00:22.339526 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.339507    2579 scope.go:117] "RemoveContainer" containerID="90b12dd49290b9a6b872a7e55e4f172b494433d87a8194303ff8939d0f437274"
Aug 04 04:00:22.339843 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:00:22.339823    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b12dd49290b9a6b872a7e55e4f172b494433d87a8194303ff8939d0f437274\": container with ID starting with 90b12dd49290b9a6b872a7e55e4f172b494433d87a8194303ff8939d0f437274 not found: ID does not exist" containerID="90b12dd49290b9a6b872a7e55e4f172b494433d87a8194303ff8939d0f437274"
Aug 04 04:00:22.339895 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.339852    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b12dd49290b9a6b872a7e55e4f172b494433d87a8194303ff8939d0f437274"} err="failed to get container status \"90b12dd49290b9a6b872a7e55e4f172b494433d87a8194303ff8939d0f437274\": rpc error: code = NotFound desc = could not find container \"90b12dd49290b9a6b872a7e55e4f172b494433d87a8194303ff8939d0f437274\": container with ID starting with 90b12dd49290b9a6b872a7e55e4f172b494433d87a8194303ff8939d0f437274 not found: ID does not exist"
Aug 04 04:00:22.341451 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.341431    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"]
Aug 04 04:00:22.343251 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.343233    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-523226-7cbd56d4f6-vp6px"]
Aug 04 04:00:22.360759 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.360730    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86"]
Aug 04 04:00:22.364027 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.364003    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-523226-cbddbf776-qrr86"]
Aug 04 04:00:22.375858 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.375833    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw"]
Aug 04 04:00:22.381763 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:22.381739    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-523226-749d6f4d4b-4bngw"]
Aug 04 04:00:23.845433 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:23.845390    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="364e00c6-b98b-47d9-9719-3e563a2ab487" path="/var/lib/kubelet/pods/364e00c6-b98b-47d9-9719-3e563a2ab487/volumes"
Aug 04 04:00:23.845989 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:23.845945    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca" path="/var/lib/kubelet/pods/7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca/volumes"
Aug 04 04:00:23.846421 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:23.846396    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0bc8f75-bb42-4be3-b3bc-456ef2318980" path="/var/lib/kubelet/pods/d0bc8f75-bb42-4be3-b3bc-456ef2318980/volumes"
Aug 04 04:00:33.500126 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.500088    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"]
Aug 04 04:00:33.500674 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.500658    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0bc8f75-bb42-4be3-b3bc-456ef2318980" containerName="payload-processing"
Aug 04 04:00:33.500726 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.500677    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0bc8f75-bb42-4be3-b3bc-456ef2318980" containerName="payload-processing"
Aug 04 04:00:33.500726 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.500707    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca" containerName="maas-api"
Aug 04 04:00:33.500726 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.500716    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca" containerName="maas-api"
Aug 04 04:00:33.500840 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.500735    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="364e00c6-b98b-47d9-9719-3e563a2ab487" containerName="payload-pre-processing"
Aug 04 04:00:33.500840 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.500745    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="364e00c6-b98b-47d9-9719-3e563a2ab487" containerName="payload-pre-processing"
Aug 04 04:00:33.500840 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.500757    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ac9d0f-3ef8-4690-964f-d7a1fe553642" containerName="payload-processing"
Aug 04 04:00:33.500840 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.500766    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ac9d0f-3ef8-4690-964f-d7a1fe553642" containerName="payload-processing"
Aug 04 04:00:33.500840 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.500777    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d62fe8c2-0569-4054-8142-45784c78a8aa" containerName="maas-api"
Aug 04 04:00:33.500840 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.500786    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d62fe8c2-0569-4054-8142-45784c78a8aa" containerName="maas-api"
Aug 04 04:00:33.501052 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.500870    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dbd272a-2aeb-413d-89d5-76f8b0d9d6ca" containerName="maas-api"
Aug 04 04:00:33.501052 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.500881    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0bc8f75-bb42-4be3-b3bc-456ef2318980" containerName="payload-processing"
Aug 04 04:00:33.501052 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.500892    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ac9d0f-3ef8-4690-964f-d7a1fe553642" containerName="payload-processing"
Aug 04 04:00:33.501052 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.500901    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="364e00c6-b98b-47d9-9719-3e563a2ab487" containerName="payload-pre-processing"
Aug 04 04:00:33.501052 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.500910    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d62fe8c2-0569-4054-8142-45784c78a8aa" containerName="maas-api"
Aug 04 04:00:33.506169 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.506148    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"
Aug 04 04:00:33.511014 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.510984    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-07c848-serving-cert\""
Aug 04 04:00:33.524405 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.524375    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"]
Aug 04 04:00:33.580766 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.580728    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m55pg\" (UniqueName: \"kubernetes.io/projected/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-kube-api-access-m55pg\") pod \"maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4\" (UID: \"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"
Aug 04 04:00:33.580766 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.580775    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4\" (UID: \"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"
Aug 04 04:00:33.581018 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.580796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-maas-api-tls\") pod \"maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4\" (UID: \"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"
Aug 04 04:00:33.682185 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.682146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m55pg\" (UniqueName: \"kubernetes.io/projected/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-kube-api-access-m55pg\") pod \"maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4\" (UID: \"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"
Aug 04 04:00:33.682362 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.682205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4\" (UID: \"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"
Aug 04 04:00:33.682362 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.682239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-maas-api-tls\") pod \"maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4\" (UID: \"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"
Aug 04 04:00:33.682441 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:00:33.682409    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-serving-cert: secret "maas-api-e2e-shared-a-07c848-serving-cert" not found
Aug 04 04:00:33.682496 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:00:33.682484    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-maas-api-tls podName:1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2 nodeName:}" failed. No retries permitted until 2026-08-04 04:00:34.182463371 +0000 UTC m=+3064.862250470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-maas-api-tls") pod "maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4" (UID: "1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2") : secret "maas-api-e2e-shared-a-07c848-serving-cert" not found
Aug 04 04:00:33.682875 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.682855    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4\" (UID: \"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"
Aug 04 04:00:33.706368 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:33.706341    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m55pg\" (UniqueName: \"kubernetes.io/projected/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-kube-api-access-m55pg\") pod \"maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4\" (UID: \"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"
Aug 04 04:00:34.188582 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:34.188539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-maas-api-tls\") pod \"maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4\" (UID: \"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"
Aug 04 04:00:34.191189 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:34.191155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-maas-api-tls\") pod \"maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4\" (UID: \"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"
Aug 04 04:00:34.276460 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:34.276422    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b"]
Aug 04 04:00:34.280209 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:34.280188    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b"
Aug 04 04:00:34.283641 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:34.283617    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-07c848\""
Aug 04 04:00:34.289171 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:34.289138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv5d9\" (UniqueName: \"kubernetes.io/projected/683fc823-1d0c-46e6-addb-31a35d0cc710-kube-api-access-gv5d9\") pod \"payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b\" (UID: \"683fc823-1d0c-46e6-addb-31a35d0cc710\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b"
Aug 04 04:00:34.289324 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:34.289289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/683fc823-1d0c-46e6-addb-31a35d0cc710-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b\" (UID: \"683fc823-1d0c-46e6-addb-31a35d0cc710\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b"
Aug 04 04:00:34.290865 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:34.290840    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b"]
Aug 04 04:00:34.390839 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:34.390802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/683fc823-1d0c-46e6-addb-31a35d0cc710-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b\" (UID: \"683fc823-1d0c-46e6-addb-31a35d0cc710\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b"
Aug 04 04:00:34.391055 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:34.390862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv5d9\" (UniqueName: \"kubernetes.io/projected/683fc823-1d0c-46e6-addb-31a35d0cc710-kube-api-access-gv5d9\") pod \"payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b\" (UID: \"683fc823-1d0c-46e6-addb-31a35d0cc710\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b"
Aug 04 04:00:34.391533 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:34.391513    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/683fc823-1d0c-46e6-addb-31a35d0cc710-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b\" (UID: \"683fc823-1d0c-46e6-addb-31a35d0cc710\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b"
Aug 04 04:00:34.400616 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:34.400589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv5d9\" (UniqueName: \"kubernetes.io/projected/683fc823-1d0c-46e6-addb-31a35d0cc710-kube-api-access-gv5d9\") pod \"payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b\" (UID: \"683fc823-1d0c-46e6-addb-31a35d0cc710\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b"
Aug 04 04:00:34.421154 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:34.421123    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"
Aug 04 04:00:34.561626 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:34.561592    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"]
Aug 04 04:00:34.565833 ip-10-0-138-99 kubenswrapper[2579]: W0804 04:00:34.565797    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd6e00c_f5d1_4aa8_b955_2c295b5f33b2.slice/crio-417eff3d8ca5ecc438439a0a65e493117789cab314d4e798d20b2a5dc10f13a0 WatchSource:0}: Error finding container 417eff3d8ca5ecc438439a0a65e493117789cab314d4e798d20b2a5dc10f13a0: Status 404 returned error can't find the container with id 417eff3d8ca5ecc438439a0a65e493117789cab314d4e798d20b2a5dc10f13a0
Aug 04 04:00:34.592764 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:34.592723    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b"
Aug 04 04:00:34.728746 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:34.728720    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b"]
Aug 04 04:00:34.730970 ip-10-0-138-99 kubenswrapper[2579]: W0804 04:00:34.730932    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683fc823_1d0c_46e6_addb_31a35d0cc710.slice/crio-246981a600d896c4d98a04cd6b25dee16aad747de948e48e76861cea2e85ae7b WatchSource:0}: Error finding container 246981a600d896c4d98a04cd6b25dee16aad747de948e48e76861cea2e85ae7b: Status 404 returned error can't find the container with id 246981a600d896c4d98a04cd6b25dee16aad747de948e48e76861cea2e85ae7b
Aug 04 04:00:35.386034 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:35.385994    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b" event={"ID":"683fc823-1d0c-46e6-addb-31a35d0cc710","Type":"ContainerStarted","Data":"ec1cfbaa5d128f1d674c190dee313807b5f74e734d264f987b167e60f5870c41"}
Aug 04 04:00:35.386202 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:35.386064    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b"
Aug 04 04:00:35.386202 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:35.386090    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b" event={"ID":"683fc823-1d0c-46e6-addb-31a35d0cc710","Type":"ContainerStarted","Data":"246981a600d896c4d98a04cd6b25dee16aad747de948e48e76861cea2e85ae7b"}
Aug 04 04:00:35.387338 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:35.387314    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4" event={"ID":"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2","Type":"ContainerStarted","Data":"611ecaefd7ba2d25bb77fb0c955ecdfd26ed75b1ca224f4d4fbd82c4c8392da4"}
Aug 04 04:00:35.387438 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:35.387354    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4" event={"ID":"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2","Type":"ContainerStarted","Data":"417eff3d8ca5ecc438439a0a65e493117789cab314d4e798d20b2a5dc10f13a0"}
Aug 04 04:00:35.387438 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:35.387375    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"
Aug 04 04:00:35.404816 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:35.404770    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b" podStartSLOduration=1.404755213 podStartE2EDuration="1.404755213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:00:35.404009354 +0000 UTC m=+3066.083796492" watchObservedRunningTime="2026-08-04 04:00:35.404755213 +0000 UTC m=+3066.084542334"
Aug 04 04:00:35.424007 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:35.423906    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4" podStartSLOduration=1.82298734 podStartE2EDuration="2.42388881s" totalImagesPullingTime="600.90147ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:00:35.422869734 +0000 UTC m=+3066.102656877" watchObservedRunningTime="2026-08-04 04:00:35.42388881 +0000 UTC m=+3066.103675929"
Aug 04 04:00:41.397709 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:41.397679    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"
Aug 04 04:00:46.394620 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:00:46.394584    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b"
Aug 04 04:01:02.672287 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:02.672251    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"]
Aug 04 04:01:02.676031 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:02.676007    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"
Aug 04 04:01:02.679584 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:02.679559    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-881c6f-serving-cert\""
Aug 04 04:01:02.712506 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:02.712478    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"]
Aug 04 04:01:02.766000 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:02.765943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/667d9804-3048-46b4-9eda-03bb5a3febbc-maas-api-tls\") pod \"maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz\" (UID: \"667d9804-3048-46b4-9eda-03bb5a3febbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"
Aug 04 04:01:02.766184 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:02.766078    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhf46\" (UniqueName: \"kubernetes.io/projected/667d9804-3048-46b4-9eda-03bb5a3febbc-kube-api-access-vhf46\") pod \"maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz\" (UID: \"667d9804-3048-46b4-9eda-03bb5a3febbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"
Aug 04 04:01:02.766184 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:02.766136    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/667d9804-3048-46b4-9eda-03bb5a3febbc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz\" (UID: \"667d9804-3048-46b4-9eda-03bb5a3febbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"
Aug 04 04:01:02.867456 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:02.867423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/667d9804-3048-46b4-9eda-03bb5a3febbc-maas-api-tls\") pod \"maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz\" (UID: \"667d9804-3048-46b4-9eda-03bb5a3febbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"
Aug 04 04:01:02.867645 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:02.867494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhf46\" (UniqueName: \"kubernetes.io/projected/667d9804-3048-46b4-9eda-03bb5a3febbc-kube-api-access-vhf46\") pod \"maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz\" (UID: \"667d9804-3048-46b4-9eda-03bb5a3febbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"
Aug 04 04:01:02.867645 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:02.867522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/667d9804-3048-46b4-9eda-03bb5a3febbc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz\" (UID: \"667d9804-3048-46b4-9eda-03bb5a3febbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"
Aug 04 04:01:02.868212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:02.868184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/667d9804-3048-46b4-9eda-03bb5a3febbc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz\" (UID: \"667d9804-3048-46b4-9eda-03bb5a3febbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"
Aug 04 04:01:02.870082 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:02.870062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/667d9804-3048-46b4-9eda-03bb5a3febbc-maas-api-tls\") pod \"maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz\" (UID: \"667d9804-3048-46b4-9eda-03bb5a3febbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"
Aug 04 04:01:02.879864 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:02.879841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhf46\" (UniqueName: \"kubernetes.io/projected/667d9804-3048-46b4-9eda-03bb5a3febbc-kube-api-access-vhf46\") pod \"maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz\" (UID: \"667d9804-3048-46b4-9eda-03bb5a3febbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"
Aug 04 04:01:02.986461 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:02.986425    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"
Aug 04 04:01:03.150092 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:03.150065    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"]
Aug 04 04:01:03.152740 ip-10-0-138-99 kubenswrapper[2579]: W0804 04:01:03.152710    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667d9804_3048_46b4_9eda_03bb5a3febbc.slice/crio-e2cebc369cfe6e0f43ea7f6642eb979c39aa0a3d72d1f923d4bd0e13a55c7ae1 WatchSource:0}: Error finding container e2cebc369cfe6e0f43ea7f6642eb979c39aa0a3d72d1f923d4bd0e13a55c7ae1: Status 404 returned error can't find the container with id e2cebc369cfe6e0f43ea7f6642eb979c39aa0a3d72d1f923d4bd0e13a55c7ae1
Aug 04 04:01:03.526679 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:03.526649    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz" event={"ID":"667d9804-3048-46b4-9eda-03bb5a3febbc","Type":"ContainerStarted","Data":"e2cebc369cfe6e0f43ea7f6642eb979c39aa0a3d72d1f923d4bd0e13a55c7ae1"}
Aug 04 04:01:03.802005 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:03.801979    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2"]
Aug 04 04:01:03.805618 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:03.805598    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2"
Aug 04 04:01:03.809625 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:03.809604    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-881c6f\""
Aug 04 04:01:03.810372 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:03.810352    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-881c6f-dockercfg-7wk88\""
Aug 04 04:01:03.832327 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:03.832297    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2"]
Aug 04 04:01:03.870608 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:03.870572    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn"]
Aug 04 04:01:03.874906 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:03.874884    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn"
Aug 04 04:01:03.889777 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:03.889750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn"]
Aug 04 04:01:03.976923 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:03.976889    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcfwx\" (UniqueName: \"kubernetes.io/projected/6ae535cf-c48a-491b-94e3-4188808e06e4-kube-api-access-vcfwx\") pod \"payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2\" (UID: \"6ae535cf-c48a-491b-94e3-4188808e06e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2"
Aug 04 04:01:03.977142 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:03.976990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe53c783-5a46-4df1-8678-3bac9b39849b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn\" (UID: \"fe53c783-5a46-4df1-8678-3bac9b39849b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn"
Aug 04 04:01:03.977142 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:03.977022    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp75n\" (UniqueName: \"kubernetes.io/projected/fe53c783-5a46-4df1-8678-3bac9b39849b-kube-api-access-wp75n\") pod \"payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn\" (UID: \"fe53c783-5a46-4df1-8678-3bac9b39849b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn"
Aug 04 04:01:03.977142 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:03.977093    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae535cf-c48a-491b-94e3-4188808e06e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2\" (UID: \"6ae535cf-c48a-491b-94e3-4188808e06e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2"
Aug 04 04:01:04.078683 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.078593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcfwx\" (UniqueName: \"kubernetes.io/projected/6ae535cf-c48a-491b-94e3-4188808e06e4-kube-api-access-vcfwx\") pod \"payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2\" (UID: \"6ae535cf-c48a-491b-94e3-4188808e06e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2"
Aug 04 04:01:04.078683 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.078664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe53c783-5a46-4df1-8678-3bac9b39849b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn\" (UID: \"fe53c783-5a46-4df1-8678-3bac9b39849b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn"
Aug 04 04:01:04.078917 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.078686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp75n\" (UniqueName: \"kubernetes.io/projected/fe53c783-5a46-4df1-8678-3bac9b39849b-kube-api-access-wp75n\") pod \"payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn\" (UID: \"fe53c783-5a46-4df1-8678-3bac9b39849b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn"
Aug 04 04:01:04.078917 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.078708    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae535cf-c48a-491b-94e3-4188808e06e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2\" (UID: \"6ae535cf-c48a-491b-94e3-4188808e06e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2"
Aug 04 04:01:04.079510 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.079483    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe53c783-5a46-4df1-8678-3bac9b39849b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn\" (UID: \"fe53c783-5a46-4df1-8678-3bac9b39849b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn"
Aug 04 04:01:04.079510 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.079503    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae535cf-c48a-491b-94e3-4188808e06e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2\" (UID: \"6ae535cf-c48a-491b-94e3-4188808e06e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2"
Aug 04 04:01:04.091498 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.091465    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcfwx\" (UniqueName: \"kubernetes.io/projected/6ae535cf-c48a-491b-94e3-4188808e06e4-kube-api-access-vcfwx\") pod \"payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2\" (UID: \"6ae535cf-c48a-491b-94e3-4188808e06e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2"
Aug 04 04:01:04.092801 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.092776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp75n\" (UniqueName: \"kubernetes.io/projected/fe53c783-5a46-4df1-8678-3bac9b39849b-kube-api-access-wp75n\") pod \"payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn\" (UID: \"fe53c783-5a46-4df1-8678-3bac9b39849b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn"
Aug 04 04:01:04.115800 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.115757    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2"
Aug 04 04:01:04.190008 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.189973    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn"
Aug 04 04:01:04.268954 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.268924    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2"]
Aug 04 04:01:04.269368 ip-10-0-138-99 kubenswrapper[2579]: W0804 04:01:04.269341    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae535cf_c48a_491b_94e3_4188808e06e4.slice/crio-d3f5ec73f7063da5d0ee5646405a732c7a027eaa621d648688f8dead3343559c WatchSource:0}: Error finding container d3f5ec73f7063da5d0ee5646405a732c7a027eaa621d648688f8dead3343559c: Status 404 returned error can't find the container with id d3f5ec73f7063da5d0ee5646405a732c7a027eaa621d648688f8dead3343559c
Aug 04 04:01:04.349003 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.348971    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn"]
Aug 04 04:01:04.349679 ip-10-0-138-99 kubenswrapper[2579]: W0804 04:01:04.349638    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe53c783_5a46_4df1_8678_3bac9b39849b.slice/crio-65d2b673cbc4785f36f8df4eae09d5604b84b554157253eb5ead80562f036241 WatchSource:0}: Error finding container 65d2b673cbc4785f36f8df4eae09d5604b84b554157253eb5ead80562f036241: Status 404 returned error can't find the container with id 65d2b673cbc4785f36f8df4eae09d5604b84b554157253eb5ead80562f036241
Aug 04 04:01:04.533934 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.533896    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz" event={"ID":"667d9804-3048-46b4-9eda-03bb5a3febbc","Type":"ContainerStarted","Data":"c8cbc4779bc9b90dd48802db4fc41273ef61a08f0b087fd3db2da3ee2756ccb5"}
Aug 04 04:01:04.534189 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.534165    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"
Aug 04 04:01:04.536206 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.536178    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn" event={"ID":"fe53c783-5a46-4df1-8678-3bac9b39849b","Type":"ContainerStarted","Data":"c0a3f27e6d11a4fd53005c275d626d18f8d5591143f25262a219883a93d23642"}
Aug 04 04:01:04.536351 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.536246    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn"
Aug 04 04:01:04.536351 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.536272    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn" event={"ID":"fe53c783-5a46-4df1-8678-3bac9b39849b","Type":"ContainerStarted","Data":"65d2b673cbc4785f36f8df4eae09d5604b84b554157253eb5ead80562f036241"}
Aug 04 04:01:04.538171 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.538135    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2" event={"ID":"6ae535cf-c48a-491b-94e3-4188808e06e4","Type":"ContainerStarted","Data":"7bc1254506ddc9a76402a18a108c6524707dab0b2f1a6513d1c1689025c68073"}
Aug 04 04:01:04.538286 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.538184    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2" event={"ID":"6ae535cf-c48a-491b-94e3-4188808e06e4","Type":"ContainerStarted","Data":"d3f5ec73f7063da5d0ee5646405a732c7a027eaa621d648688f8dead3343559c"}
Aug 04 04:01:04.538286 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.538254    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2"
Aug 04 04:01:04.559571 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.559511    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz" podStartSLOduration=1.977815632 podStartE2EDuration="2.559495063s" totalImagesPullingTime="581.679431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:01:04.557115285 +0000 UTC m=+3095.236902423" watchObservedRunningTime="2026-08-04 04:01:04.559495063 +0000 UTC m=+3095.239282181"
Aug 04 04:01:04.574542 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.574489    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2" podStartSLOduration=1.5744749 podStartE2EDuration="1.5744749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:01:04.573170452 +0000 UTC m=+3095.252957613" watchObservedRunningTime="2026-08-04 04:01:04.5744749 +0000 UTC m=+3095.254262020"
Aug 04 04:01:04.592130 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:04.592027    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn" podStartSLOduration=1.592009577 podStartE2EDuration="1.592009577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:01:04.590716796 +0000 UTC m=+3095.270503935" watchObservedRunningTime="2026-08-04 04:01:04.592009577 +0000 UTC m=+3095.271796696"
Aug 04 04:01:10.549423 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:10.549392    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"
Aug 04 04:01:15.545227 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:15.545189    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn"
Aug 04 04:01:15.545674 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:01:15.545501    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2"
Aug 04 04:11:53.185634 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:53.185600    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7436-140134711430448"]
Aug 04 04:11:53.189460 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:53.189437    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711430448"
Aug 04 04:11:53.192102 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:53.192081    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-tdvwx\""
Aug 04 04:11:53.193930 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:53.193906    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7436-140134711430448"]
Aug 04 04:11:53.247300 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:53.247263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdt6f\" (UniqueName: \"kubernetes.io/projected/ab06baad-1a8a-4221-b400-4f84f47faf21-kube-api-access-kdt6f\") pod \"test-curl-7436-140134711430448\" (UID: \"ab06baad-1a8a-4221-b400-4f84f47faf21\") " pod="opendatahub/test-curl-7436-140134711430448"
Aug 04 04:11:53.348641 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:53.348592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdt6f\" (UniqueName: \"kubernetes.io/projected/ab06baad-1a8a-4221-b400-4f84f47faf21-kube-api-access-kdt6f\") pod \"test-curl-7436-140134711430448\" (UID: \"ab06baad-1a8a-4221-b400-4f84f47faf21\") " pod="opendatahub/test-curl-7436-140134711430448"
Aug 04 04:11:53.358160 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:53.358129    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdt6f\" (UniqueName: \"kubernetes.io/projected/ab06baad-1a8a-4221-b400-4f84f47faf21-kube-api-access-kdt6f\") pod \"test-curl-7436-140134711430448\" (UID: \"ab06baad-1a8a-4221-b400-4f84f47faf21\") " pod="opendatahub/test-curl-7436-140134711430448"
Aug 04 04:11:53.501200 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:53.501148    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711430448"
Aug 04 04:11:53.637851 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:53.637828    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7436-140134711430448"]
Aug 04 04:11:53.640179 ip-10-0-138-99 kubenswrapper[2579]: W0804 04:11:53.640152    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab06baad_1a8a_4221_b400_4f84f47faf21.slice/crio-732fb3296e849cc7c631a66f34a49f62517e1a033eea3685ed0cab1b0ca1763f WatchSource:0}: Error finding container 732fb3296e849cc7c631a66f34a49f62517e1a033eea3685ed0cab1b0ca1763f: Status 404 returned error can't find the container with id 732fb3296e849cc7c631a66f34a49f62517e1a033eea3685ed0cab1b0ca1763f
Aug 04 04:11:53.641564 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:53.641544    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 04:11:53.855484 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:53.855402    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7436-140134711430448" event={"ID":"ab06baad-1a8a-4221-b400-4f84f47faf21","Type":"ContainerStarted","Data":"732fb3296e849cc7c631a66f34a49f62517e1a033eea3685ed0cab1b0ca1763f"}
Aug 04 04:11:54.862320 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:54.862278    2579 generic.go:356] "Generic (PLEG): container finished" podID="ab06baad-1a8a-4221-b400-4f84f47faf21" containerID="87006ea6c81726bfe844c6bf7dfb697b06e20563d91d4ad134898619e8d6358a" exitCode=0
Aug 04 04:11:54.862714 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:54.862375    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7436-140134711430448" event={"ID":"ab06baad-1a8a-4221-b400-4f84f47faf21","Type":"ContainerDied","Data":"87006ea6c81726bfe844c6bf7dfb697b06e20563d91d4ad134898619e8d6358a"}
Aug 04 04:11:55.998735 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:55.998710    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711430448"
Aug 04 04:11:56.051793 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.051752    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7436-140134711430448_ab06baad-1a8a-4221-b400-4f84f47faf21/test-curl-7436-140134711430448/0.log"
Aug 04 04:11:56.075307 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.075272    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdt6f\" (UniqueName: \"kubernetes.io/projected/ab06baad-1a8a-4221-b400-4f84f47faf21-kube-api-access-kdt6f\") pod \"ab06baad-1a8a-4221-b400-4f84f47faf21\" (UID: \"ab06baad-1a8a-4221-b400-4f84f47faf21\") "
Aug 04 04:11:56.077659 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.077625    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab06baad-1a8a-4221-b400-4f84f47faf21-kube-api-access-kdt6f" (OuterVolumeSpecName: "kube-api-access-kdt6f") pod "ab06baad-1a8a-4221-b400-4f84f47faf21" (UID: "ab06baad-1a8a-4221-b400-4f84f47faf21"). InnerVolumeSpecName "kube-api-access-kdt6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:11:56.078572 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.078546    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7436-140134711430448"]
Aug 04 04:11:56.082520 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.082499    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7436-140134711430448"]
Aug 04 04:11:56.177032 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.176907    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdt6f\" (UniqueName: \"kubernetes.io/projected/ab06baad-1a8a-4221-b400-4f84f47faf21-kube-api-access-kdt6f\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:11:56.214955 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.214920    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7436-140134711312944"]
Aug 04 04:11:56.215399 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.215384    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab06baad-1a8a-4221-b400-4f84f47faf21" containerName="test-curl-7436-140134711430448"
Aug 04 04:11:56.215443 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.215400    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab06baad-1a8a-4221-b400-4f84f47faf21" containerName="test-curl-7436-140134711430448"
Aug 04 04:11:56.215487 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.215477    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab06baad-1a8a-4221-b400-4f84f47faf21" containerName="test-curl-7436-140134711430448"
Aug 04 04:11:56.218549 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.218534    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711312944"
Aug 04 04:11:56.227795 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.227767    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7436-140134711312944"]
Aug 04 04:11:56.277646 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.277610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cpc6\" (UniqueName: \"kubernetes.io/projected/6f951eb6-84a8-466b-b541-a3b1ae9f5d23-kube-api-access-7cpc6\") pod \"test-curl-7436-140134711312944\" (UID: \"6f951eb6-84a8-466b-b541-a3b1ae9f5d23\") " pod="opendatahub/test-curl-7436-140134711312944"
Aug 04 04:11:56.379092 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.379051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cpc6\" (UniqueName: \"kubernetes.io/projected/6f951eb6-84a8-466b-b541-a3b1ae9f5d23-kube-api-access-7cpc6\") pod \"test-curl-7436-140134711312944\" (UID: \"6f951eb6-84a8-466b-b541-a3b1ae9f5d23\") " pod="opendatahub/test-curl-7436-140134711312944"
Aug 04 04:11:56.389570 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.389544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cpc6\" (UniqueName: \"kubernetes.io/projected/6f951eb6-84a8-466b-b541-a3b1ae9f5d23-kube-api-access-7cpc6\") pod \"test-curl-7436-140134711312944\" (UID: \"6f951eb6-84a8-466b-b541-a3b1ae9f5d23\") " pod="opendatahub/test-curl-7436-140134711312944"
Aug 04 04:11:56.528795 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.528756    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711312944"
Aug 04 04:11:56.668060 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.668034    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7436-140134711312944"]
Aug 04 04:11:56.669872 ip-10-0-138-99 kubenswrapper[2579]: W0804 04:11:56.669848    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f951eb6_84a8_466b_b541_a3b1ae9f5d23.slice/crio-a14044d5909e79e82ee8dbf177988a796729ef326a2b6803eae8d3c6c5c9ea63 WatchSource:0}: Error finding container a14044d5909e79e82ee8dbf177988a796729ef326a2b6803eae8d3c6c5c9ea63: Status 404 returned error can't find the container with id a14044d5909e79e82ee8dbf177988a796729ef326a2b6803eae8d3c6c5c9ea63
Aug 04 04:11:56.874029 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.873924    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711430448"
Aug 04 04:11:56.874029 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.873930    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="732fb3296e849cc7c631a66f34a49f62517e1a033eea3685ed0cab1b0ca1763f"
Aug 04 04:11:56.875068 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:56.875045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7436-140134711312944" event={"ID":"6f951eb6-84a8-466b-b541-a3b1ae9f5d23","Type":"ContainerStarted","Data":"a14044d5909e79e82ee8dbf177988a796729ef326a2b6803eae8d3c6c5c9ea63"}
Aug 04 04:11:56.884347 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:11:56.884320    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7436-140134711430448\" is forbidden: User \"system:node:ip-10-0-138-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-99.ec2.internal' and this object" podUID="ab06baad-1a8a-4221-b400-4f84f47faf21" pod="opendatahub/test-curl-7436-140134711430448"
Aug 04 04:11:57.845811 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:57.845774    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab06baad-1a8a-4221-b400-4f84f47faf21" path="/var/lib/kubelet/pods/ab06baad-1a8a-4221-b400-4f84f47faf21/volumes"
Aug 04 04:11:57.881346 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:57.881309    2579 generic.go:356] "Generic (PLEG): container finished" podID="6f951eb6-84a8-466b-b541-a3b1ae9f5d23" containerID="f2aaa47630d26cd4e3025e6102e08bc96e9cbaf3e8b0cb24be03df86af54c8e9" exitCode=0
Aug 04 04:11:57.881483 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:57.881400    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7436-140134711312944" event={"ID":"6f951eb6-84a8-466b-b541-a3b1ae9f5d23","Type":"ContainerDied","Data":"f2aaa47630d26cd4e3025e6102e08bc96e9cbaf3e8b0cb24be03df86af54c8e9"}
Aug 04 04:11:59.031556 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.031532    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711312944"
Aug 04 04:11:59.053273 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.053240    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7436-140134711312944_6f951eb6-84a8-466b-b541-a3b1ae9f5d23/test-curl-7436-140134711312944/0.log"
Aug 04 04:11:59.075988 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.075942    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7436-140134711312944"]
Aug 04 04:11:59.079499 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.079468    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7436-140134711312944"]
Aug 04 04:11:59.106774 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.106739    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cpc6\" (UniqueName: \"kubernetes.io/projected/6f951eb6-84a8-466b-b541-a3b1ae9f5d23-kube-api-access-7cpc6\") pod \"6f951eb6-84a8-466b-b541-a3b1ae9f5d23\" (UID: \"6f951eb6-84a8-466b-b541-a3b1ae9f5d23\") "
Aug 04 04:11:59.108969 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.108939    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f951eb6-84a8-466b-b541-a3b1ae9f5d23-kube-api-access-7cpc6" (OuterVolumeSpecName: "kube-api-access-7cpc6") pod "6f951eb6-84a8-466b-b541-a3b1ae9f5d23" (UID: "6f951eb6-84a8-466b-b541-a3b1ae9f5d23"). InnerVolumeSpecName "kube-api-access-7cpc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:11:59.208004 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.207891    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cpc6\" (UniqueName: \"kubernetes.io/projected/6f951eb6-84a8-466b-b541-a3b1ae9f5d23-kube-api-access-7cpc6\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:11:59.239751 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.239716    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7436-140134711364752"]
Aug 04 04:11:59.240166 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.240151    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f951eb6-84a8-466b-b541-a3b1ae9f5d23" containerName="test-curl-7436-140134711312944"
Aug 04 04:11:59.240223 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.240168    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f951eb6-84a8-466b-b541-a3b1ae9f5d23" containerName="test-curl-7436-140134711312944"
Aug 04 04:11:59.240267 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.240239    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f951eb6-84a8-466b-b541-a3b1ae9f5d23" containerName="test-curl-7436-140134711312944"
Aug 04 04:11:59.243217 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.243199    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711364752"
Aug 04 04:11:59.249593 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.249569    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7436-140134711364752"]
Aug 04 04:11:59.308668 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.308634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg4x7\" (UniqueName: \"kubernetes.io/projected/bbd367dd-2d2f-498f-9bb8-a274eebef072-kube-api-access-rg4x7\") pod \"test-curl-7436-140134711364752\" (UID: \"bbd367dd-2d2f-498f-9bb8-a274eebef072\") " pod="opendatahub/test-curl-7436-140134711364752"
Aug 04 04:11:59.410187 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.410149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg4x7\" (UniqueName: \"kubernetes.io/projected/bbd367dd-2d2f-498f-9bb8-a274eebef072-kube-api-access-rg4x7\") pod \"test-curl-7436-140134711364752\" (UID: \"bbd367dd-2d2f-498f-9bb8-a274eebef072\") " pod="opendatahub/test-curl-7436-140134711364752"
Aug 04 04:11:59.419210 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.419169    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg4x7\" (UniqueName: \"kubernetes.io/projected/bbd367dd-2d2f-498f-9bb8-a274eebef072-kube-api-access-rg4x7\") pod \"test-curl-7436-140134711364752\" (UID: \"bbd367dd-2d2f-498f-9bb8-a274eebef072\") " pod="opendatahub/test-curl-7436-140134711364752"
Aug 04 04:11:59.554610 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.554573    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711364752"
Aug 04 04:11:59.694203 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.694169    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7436-140134711364752"]
Aug 04 04:11:59.696181 ip-10-0-138-99 kubenswrapper[2579]: W0804 04:11:59.696147    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbd367dd_2d2f_498f_9bb8_a274eebef072.slice/crio-9c0b6f6b23e8fe90ba9ac33c82431cd25367252f25a900536b20a5b59fd5f365 WatchSource:0}: Error finding container 9c0b6f6b23e8fe90ba9ac33c82431cd25367252f25a900536b20a5b59fd5f365: Status 404 returned error can't find the container with id 9c0b6f6b23e8fe90ba9ac33c82431cd25367252f25a900536b20a5b59fd5f365
Aug 04 04:11:59.845584 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.845507    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f951eb6-84a8-466b-b541-a3b1ae9f5d23" path="/var/lib/kubelet/pods/6f951eb6-84a8-466b-b541-a3b1ae9f5d23/volumes"
Aug 04 04:11:59.892955 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.892920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7436-140134711364752" event={"ID":"bbd367dd-2d2f-498f-9bb8-a274eebef072","Type":"ContainerStarted","Data":"9c0b6f6b23e8fe90ba9ac33c82431cd25367252f25a900536b20a5b59fd5f365"}
Aug 04 04:11:59.894253 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.894234    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711312944"
Aug 04 04:11:59.894253 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:11:59.894246    2579 scope.go:117] "RemoveContainer" containerID="f2aaa47630d26cd4e3025e6102e08bc96e9cbaf3e8b0cb24be03df86af54c8e9"
Aug 04 04:12:00.901703 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:00.901666    2579 generic.go:356] "Generic (PLEG): container finished" podID="bbd367dd-2d2f-498f-9bb8-a274eebef072" containerID="fb44c584fcadef5557c34e8be8138e19731cb5aabe36a0f0554dc26215697907" exitCode=0
Aug 04 04:12:00.902087 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:00.901722    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7436-140134711364752" event={"ID":"bbd367dd-2d2f-498f-9bb8-a274eebef072","Type":"ContainerDied","Data":"fb44c584fcadef5557c34e8be8138e19731cb5aabe36a0f0554dc26215697907"}
Aug 04 04:12:02.054005 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.053977    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711364752"
Aug 04 04:12:02.077894 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.077862    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7436-140134711364752_bbd367dd-2d2f-498f-9bb8-a274eebef072/test-curl-7436-140134711364752/0.log"
Aug 04 04:12:02.100679 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.100645    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7436-140134711364752"]
Aug 04 04:12:02.105177 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.105147    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7436-140134711364752"]
Aug 04 04:12:02.139493 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.139458    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rg4x7\" (UniqueName: \"kubernetes.io/projected/bbd367dd-2d2f-498f-9bb8-a274eebef072-kube-api-access-rg4x7\") pod \"bbd367dd-2d2f-498f-9bb8-a274eebef072\" (UID: \"bbd367dd-2d2f-498f-9bb8-a274eebef072\") "
Aug 04 04:12:02.141645 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.141612    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd367dd-2d2f-498f-9bb8-a274eebef072-kube-api-access-rg4x7" (OuterVolumeSpecName: "kube-api-access-rg4x7") pod "bbd367dd-2d2f-498f-9bb8-a274eebef072" (UID: "bbd367dd-2d2f-498f-9bb8-a274eebef072"). InnerVolumeSpecName "kube-api-access-rg4x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:12:02.222719 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.222685    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7436-140134711364752"]
Aug 04 04:12:02.223154 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.223128    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbd367dd-2d2f-498f-9bb8-a274eebef072" containerName="test-curl-7436-140134711364752"
Aug 04 04:12:02.223154 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.223156    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbd367dd-2d2f-498f-9bb8-a274eebef072" containerName="test-curl-7436-140134711364752"
Aug 04 04:12:02.223246 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.223227    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbd367dd-2d2f-498f-9bb8-a274eebef072" containerName="test-curl-7436-140134711364752"
Aug 04 04:12:02.226239 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.226223    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711364752"
Aug 04 04:12:02.236512 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.236487    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7436-140134711364752"]
Aug 04 04:12:02.240761 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.240729    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg4x7\" (UniqueName: \"kubernetes.io/projected/bbd367dd-2d2f-498f-9bb8-a274eebef072-kube-api-access-rg4x7\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:12:02.342072 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.342031    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g2lh\" (UniqueName: \"kubernetes.io/projected/122ec5ec-ef15-4856-9518-3b33614f994d-kube-api-access-4g2lh\") pod \"test-curl-7436-140134711364752\" (UID: \"122ec5ec-ef15-4856-9518-3b33614f994d\") " pod="opendatahub/test-curl-7436-140134711364752"
Aug 04 04:12:02.443335 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.443299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g2lh\" (UniqueName: \"kubernetes.io/projected/122ec5ec-ef15-4856-9518-3b33614f994d-kube-api-access-4g2lh\") pod \"test-curl-7436-140134711364752\" (UID: \"122ec5ec-ef15-4856-9518-3b33614f994d\") " pod="opendatahub/test-curl-7436-140134711364752"
Aug 04 04:12:02.452326 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.452300    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g2lh\" (UniqueName: \"kubernetes.io/projected/122ec5ec-ef15-4856-9518-3b33614f994d-kube-api-access-4g2lh\") pod \"test-curl-7436-140134711364752\" (UID: \"122ec5ec-ef15-4856-9518-3b33614f994d\") " pod="opendatahub/test-curl-7436-140134711364752"
Aug 04 04:12:02.535606 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.535521    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711364752"
Aug 04 04:12:02.673363 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.673336    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7436-140134711364752"]
Aug 04 04:12:02.675492 ip-10-0-138-99 kubenswrapper[2579]: W0804 04:12:02.675461    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod122ec5ec_ef15_4856_9518_3b33614f994d.slice/crio-9b6d7f8f6d30e75e4fc5bc19dadf1ca8b5259204f7e05999320360751d92f135 WatchSource:0}: Error finding container 9b6d7f8f6d30e75e4fc5bc19dadf1ca8b5259204f7e05999320360751d92f135: Status 404 returned error can't find the container with id 9b6d7f8f6d30e75e4fc5bc19dadf1ca8b5259204f7e05999320360751d92f135
Aug 04 04:12:02.914279 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.914208    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711364752"
Aug 04 04:12:02.914279 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.914228    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c0b6f6b23e8fe90ba9ac33c82431cd25367252f25a900536b20a5b59fd5f365"
Aug 04 04:12:02.915324 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.915301    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7436-140134711364752" event={"ID":"122ec5ec-ef15-4856-9518-3b33614f994d","Type":"ContainerStarted","Data":"9b6d7f8f6d30e75e4fc5bc19dadf1ca8b5259204f7e05999320360751d92f135"}
Aug 04 04:12:02.926660 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:02.926607    2579 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7436-140134711364752" oldPodUID="bbd367dd-2d2f-498f-9bb8-a274eebef072" podUID="122ec5ec-ef15-4856-9518-3b33614f994d"
Aug 04 04:12:03.846030 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:03.845993    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbd367dd-2d2f-498f-9bb8-a274eebef072" path="/var/lib/kubelet/pods/bbd367dd-2d2f-498f-9bb8-a274eebef072/volumes"
Aug 04 04:12:03.924490 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:03.924454    2579 generic.go:356] "Generic (PLEG): container finished" podID="122ec5ec-ef15-4856-9518-3b33614f994d" containerID="56132b2685c6b43fb1c156d66db39094ca21441fa54b98e0508f4532b5334f6a" exitCode=0
Aug 04 04:12:03.924681 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:03.924553    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7436-140134711364752" event={"ID":"122ec5ec-ef15-4856-9518-3b33614f994d","Type":"ContainerDied","Data":"56132b2685c6b43fb1c156d66db39094ca21441fa54b98e0508f4532b5334f6a"}
Aug 04 04:12:05.072889 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.072865    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711364752"
Aug 04 04:12:05.093918 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.093820    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7436-140134711364752_122ec5ec-ef15-4856-9518-3b33614f994d/test-curl-7436-140134711364752/0.log"
Aug 04 04:12:05.118301 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.118268    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7436-140134711364752"]
Aug 04 04:12:05.124045 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.124021    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7436-140134711364752"]
Aug 04 04:12:05.170567 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.170539    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g2lh\" (UniqueName: \"kubernetes.io/projected/122ec5ec-ef15-4856-9518-3b33614f994d-kube-api-access-4g2lh\") pod \"122ec5ec-ef15-4856-9518-3b33614f994d\" (UID: \"122ec5ec-ef15-4856-9518-3b33614f994d\") "
Aug 04 04:12:05.172589 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.172563    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122ec5ec-ef15-4856-9518-3b33614f994d-kube-api-access-4g2lh" (OuterVolumeSpecName: "kube-api-access-4g2lh") pod "122ec5ec-ef15-4856-9518-3b33614f994d" (UID: "122ec5ec-ef15-4856-9518-3b33614f994d"). InnerVolumeSpecName "kube-api-access-4g2lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:12:05.249189 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.249160    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7436-140134711259184"]
Aug 04 04:12:05.249634 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.249618    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="122ec5ec-ef15-4856-9518-3b33614f994d" containerName="test-curl-7436-140134711364752"
Aug 04 04:12:05.249677 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.249637    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="122ec5ec-ef15-4856-9518-3b33614f994d" containerName="test-curl-7436-140134711364752"
Aug 04 04:12:05.249722 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.249712    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="122ec5ec-ef15-4856-9518-3b33614f994d" containerName="test-curl-7436-140134711364752"
Aug 04 04:12:05.252649 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.252633    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711259184"
Aug 04 04:12:05.260240 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.260200    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7436-140134711259184"]
Aug 04 04:12:05.272052 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.272020    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g2lh\" (UniqueName: \"kubernetes.io/projected/122ec5ec-ef15-4856-9518-3b33614f994d-kube-api-access-4g2lh\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:12:05.373247 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.373210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvdwh\" (UniqueName: \"kubernetes.io/projected/624017bc-ab14-4858-9966-1521bd9bd8bb-kube-api-access-hvdwh\") pod \"test-curl-7436-140134711259184\" (UID: \"624017bc-ab14-4858-9966-1521bd9bd8bb\") " pod="opendatahub/test-curl-7436-140134711259184"
Aug 04 04:12:05.474122 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.474082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdwh\" (UniqueName: \"kubernetes.io/projected/624017bc-ab14-4858-9966-1521bd9bd8bb-kube-api-access-hvdwh\") pod \"test-curl-7436-140134711259184\" (UID: \"624017bc-ab14-4858-9966-1521bd9bd8bb\") " pod="opendatahub/test-curl-7436-140134711259184"
Aug 04 04:12:05.482510 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.482482    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvdwh\" (UniqueName: \"kubernetes.io/projected/624017bc-ab14-4858-9966-1521bd9bd8bb-kube-api-access-hvdwh\") pod \"test-curl-7436-140134711259184\" (UID: \"624017bc-ab14-4858-9966-1521bd9bd8bb\") " pod="opendatahub/test-curl-7436-140134711259184"
Aug 04 04:12:05.563499 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.563398    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711259184"
Aug 04 04:12:05.704916 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.704884    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7436-140134711259184"]
Aug 04 04:12:05.706864 ip-10-0-138-99 kubenswrapper[2579]: W0804 04:12:05.706838    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod624017bc_ab14_4858_9966_1521bd9bd8bb.slice/crio-ca9d498e802ce191e82e3d63a44424b6f9c61a1d12e3e3848ea1f763a5617b6e WatchSource:0}: Error finding container ca9d498e802ce191e82e3d63a44424b6f9c61a1d12e3e3848ea1f763a5617b6e: Status 404 returned error can't find the container with id ca9d498e802ce191e82e3d63a44424b6f9c61a1d12e3e3848ea1f763a5617b6e
Aug 04 04:12:05.845574 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.845492    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="122ec5ec-ef15-4856-9518-3b33614f994d" path="/var/lib/kubelet/pods/122ec5ec-ef15-4856-9518-3b33614f994d/volumes"
Aug 04 04:12:05.936740 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.936706    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7436-140134711259184" event={"ID":"624017bc-ab14-4858-9966-1521bd9bd8bb","Type":"ContainerStarted","Data":"ca9d498e802ce191e82e3d63a44424b6f9c61a1d12e3e3848ea1f763a5617b6e"}
Aug 04 04:12:05.938001 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.937980    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711364752"
Aug 04 04:12:05.938001 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:05.937997    2579 scope.go:117] "RemoveContainer" containerID="56132b2685c6b43fb1c156d66db39094ca21441fa54b98e0508f4532b5334f6a"
Aug 04 04:12:06.947401 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:06.947282    2579 generic.go:356] "Generic (PLEG): container finished" podID="624017bc-ab14-4858-9966-1521bd9bd8bb" containerID="b7a8e945903917a8ffd969ec5611a96f71b7417ab0732ac340a32fce5ed64e59" exitCode=0
Aug 04 04:12:06.947401 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:06.947334    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7436-140134711259184" event={"ID":"624017bc-ab14-4858-9966-1521bd9bd8bb","Type":"ContainerDied","Data":"b7a8e945903917a8ffd969ec5611a96f71b7417ab0732ac340a32fce5ed64e59"}
Aug 04 04:12:08.094862 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:08.094837    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711259184"
Aug 04 04:12:08.116755 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:08.116721    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7436-140134711259184_624017bc-ab14-4858-9966-1521bd9bd8bb/test-curl-7436-140134711259184/0.log"
Aug 04 04:12:08.141245 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:08.141205    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7436-140134711259184"]
Aug 04 04:12:08.144405 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:08.144383    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7436-140134711259184"]
Aug 04 04:12:08.203200 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:08.203118    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvdwh\" (UniqueName: \"kubernetes.io/projected/624017bc-ab14-4858-9966-1521bd9bd8bb-kube-api-access-hvdwh\") pod \"624017bc-ab14-4858-9966-1521bd9bd8bb\" (UID: \"624017bc-ab14-4858-9966-1521bd9bd8bb\") "
Aug 04 04:12:08.205179 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:08.205148    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/624017bc-ab14-4858-9966-1521bd9bd8bb-kube-api-access-hvdwh" (OuterVolumeSpecName: "kube-api-access-hvdwh") pod "624017bc-ab14-4858-9966-1521bd9bd8bb" (UID: "624017bc-ab14-4858-9966-1521bd9bd8bb"). InnerVolumeSpecName "kube-api-access-hvdwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:12:08.304854 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:08.304811    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvdwh\" (UniqueName: \"kubernetes.io/projected/624017bc-ab14-4858-9966-1521bd9bd8bb-kube-api-access-hvdwh\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:12:08.959630 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:08.959604    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca9d498e802ce191e82e3d63a44424b6f9c61a1d12e3e3848ea1f763a5617b6e"
Aug 04 04:12:08.959813 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:08.959636    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-140134711259184"
Aug 04 04:12:08.969526 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:12:08.969498    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7436-140134711259184\" is forbidden: User \"system:node:ip-10-0-138-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-99.ec2.internal' and this object" podUID="624017bc-ab14-4858-9966-1521bd9bd8bb" pod="opendatahub/test-curl-7436-140134711259184"
Aug 04 04:12:09.843614 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:12:09.843578    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7436-140134711259184\" is forbidden: User \"system:node:ip-10-0-138-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-99.ec2.internal' and this object" podUID="624017bc-ab14-4858-9966-1521bd9bd8bb" pod="opendatahub/test-curl-7436-140134711259184"
Aug 04 04:12:09.845268 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:09.845246    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="624017bc-ab14-4858-9966-1521bd9bd8bb" path="/var/lib/kubelet/pods/624017bc-ab14-4858-9966-1521bd9bd8bb/volumes"
Aug 04 04:12:16.055261 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.055222    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"]
Aug 04 04:12:16.056007 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.055949    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="624017bc-ab14-4858-9966-1521bd9bd8bb" containerName="test-curl-7436-140134711259184"
Aug 04 04:12:16.056007 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.056007    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="624017bc-ab14-4858-9966-1521bd9bd8bb" containerName="test-curl-7436-140134711259184"
Aug 04 04:12:16.056174 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.056088    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="624017bc-ab14-4858-9966-1521bd9bd8bb" containerName="test-curl-7436-140134711259184"
Aug 04 04:12:16.059134 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.059114    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"
Aug 04 04:12:16.061628 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.061606    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2d8d0c-serving-cert\""
Aug 04 04:12:16.069565 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.069543    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"]
Aug 04 04:12:16.177043 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.176997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5\" (UID: \"4e57d680-e609-4c20-ac5e-9bb7ae9bf560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"
Aug 04 04:12:16.177234 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.177123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hf27\" (UniqueName: \"kubernetes.io/projected/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-kube-api-access-6hf27\") pod \"maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5\" (UID: \"4e57d680-e609-4c20-ac5e-9bb7ae9bf560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"
Aug 04 04:12:16.177234 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.177214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5\" (UID: \"4e57d680-e609-4c20-ac5e-9bb7ae9bf560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"
Aug 04 04:12:16.278662 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.278620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5\" (UID: \"4e57d680-e609-4c20-ac5e-9bb7ae9bf560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"
Aug 04 04:12:16.278845 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.278718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hf27\" (UniqueName: \"kubernetes.io/projected/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-kube-api-access-6hf27\") pod \"maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5\" (UID: \"4e57d680-e609-4c20-ac5e-9bb7ae9bf560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"
Aug 04 04:12:16.278845 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.278761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5\" (UID: \"4e57d680-e609-4c20-ac5e-9bb7ae9bf560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"
Aug 04 04:12:16.279033 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:12:16.278885    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-serving-cert: secret "maas-api-e2e-ipp-a-2d8d0c-serving-cert" not found
Aug 04 04:12:16.279033 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:12:16.278989    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-maas-api-tls podName:4e57d680-e609-4c20-ac5e-9bb7ae9bf560 nodeName:}" failed. No retries permitted until 2026-08-04 04:12:16.778935006 +0000 UTC m=+3767.458722118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-maas-api-tls") pod "maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5" (UID: "4e57d680-e609-4c20-ac5e-9bb7ae9bf560") : secret "maas-api-e2e-ipp-a-2d8d0c-serving-cert" not found
Aug 04 04:12:16.279460 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.279438    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5\" (UID: \"4e57d680-e609-4c20-ac5e-9bb7ae9bf560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"
Aug 04 04:12:16.288429 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.288402    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hf27\" (UniqueName: \"kubernetes.io/projected/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-kube-api-access-6hf27\") pod \"maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5\" (UID: \"4e57d680-e609-4c20-ac5e-9bb7ae9bf560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"
Aug 04 04:12:16.785470 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.785430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5\" (UID: \"4e57d680-e609-4c20-ac5e-9bb7ae9bf560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"
Aug 04 04:12:16.787881 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.787861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5\" (UID: \"4e57d680-e609-4c20-ac5e-9bb7ae9bf560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"
Aug 04 04:12:16.843198 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.843157    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx"]
Aug 04 04:12:16.846814 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.846791    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx"
Aug 04 04:12:16.850331 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.850306    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2d8d0c\""
Aug 04 04:12:16.850463 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.850366    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2d8d0c-dockercfg-ns8k9\""
Aug 04 04:12:16.860059 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.860033    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx"]
Aug 04 04:12:16.879143 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.879109    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb"]
Aug 04 04:12:16.886040 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.885224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb"
Aug 04 04:12:16.893484 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.893450    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb"]
Aug 04 04:12:16.973101 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.973056    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"
Aug 04 04:12:16.988575 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.988534    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j8bp\" (UniqueName: \"kubernetes.io/projected/4cdb62bc-26d6-42dd-ab6c-93c462d537b2-kube-api-access-6j8bp\") pod \"payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx\" (UID: \"4cdb62bc-26d6-42dd-ab6c-93c462d537b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx"
Aug 04 04:12:16.988742 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.988611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvj75\" (UniqueName: \"kubernetes.io/projected/e76c67ae-9ab0-4e3f-990d-e5a11a736e21-kube-api-access-rvj75\") pod \"payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb\" (UID: \"e76c67ae-9ab0-4e3f-990d-e5a11a736e21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb"
Aug 04 04:12:16.988742 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.988685    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76c67ae-9ab0-4e3f-990d-e5a11a736e21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb\" (UID: \"e76c67ae-9ab0-4e3f-990d-e5a11a736e21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb"
Aug 04 04:12:16.988939 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:16.988798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cdb62bc-26d6-42dd-ab6c-93c462d537b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx\" (UID: \"4cdb62bc-26d6-42dd-ab6c-93c462d537b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx"
Aug 04 04:12:17.089692 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:17.089642    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76c67ae-9ab0-4e3f-990d-e5a11a736e21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb\" (UID: \"e76c67ae-9ab0-4e3f-990d-e5a11a736e21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb"
Aug 04 04:12:17.090187 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:17.089773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cdb62bc-26d6-42dd-ab6c-93c462d537b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx\" (UID: \"4cdb62bc-26d6-42dd-ab6c-93c462d537b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx"
Aug 04 04:12:17.090187 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:17.089879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j8bp\" (UniqueName: \"kubernetes.io/projected/4cdb62bc-26d6-42dd-ab6c-93c462d537b2-kube-api-access-6j8bp\") pod \"payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx\" (UID: \"4cdb62bc-26d6-42dd-ab6c-93c462d537b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx"
Aug 04 04:12:17.090187 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:17.089919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvj75\" (UniqueName: \"kubernetes.io/projected/e76c67ae-9ab0-4e3f-990d-e5a11a736e21-kube-api-access-rvj75\") pod \"payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb\" (UID: \"e76c67ae-9ab0-4e3f-990d-e5a11a736e21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb"
Aug 04 04:12:17.090548 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:17.090526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cdb62bc-26d6-42dd-ab6c-93c462d537b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx\" (UID: \"4cdb62bc-26d6-42dd-ab6c-93c462d537b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx"
Aug 04 04:12:17.090627 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:17.090560    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76c67ae-9ab0-4e3f-990d-e5a11a736e21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb\" (UID: \"e76c67ae-9ab0-4e3f-990d-e5a11a736e21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb"
Aug 04 04:12:17.102344 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:17.102284    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvj75\" (UniqueName: \"kubernetes.io/projected/e76c67ae-9ab0-4e3f-990d-e5a11a736e21-kube-api-access-rvj75\") pod \"payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb\" (UID: \"e76c67ae-9ab0-4e3f-990d-e5a11a736e21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb"
Aug 04 04:12:17.102729 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:17.102704    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j8bp\" (UniqueName: \"kubernetes.io/projected/4cdb62bc-26d6-42dd-ab6c-93c462d537b2-kube-api-access-6j8bp\") pod \"payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx\" (UID: \"4cdb62bc-26d6-42dd-ab6c-93c462d537b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx"
Aug 04 04:12:17.121115 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:17.121087    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"]
Aug 04 04:12:17.122219 ip-10-0-138-99 kubenswrapper[2579]: W0804 04:12:17.122191    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e57d680_e609_4c20_ac5e_9bb7ae9bf560.slice/crio-9d3e5812a065b552381d43e3c62f779be1545525e948b0801403177527084a7f WatchSource:0}: Error finding container 9d3e5812a065b552381d43e3c62f779be1545525e948b0801403177527084a7f: Status 404 returned error can't find the container with id 9d3e5812a065b552381d43e3c62f779be1545525e948b0801403177527084a7f
Aug 04 04:12:17.158291 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:17.158264    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx"
Aug 04 04:12:17.202289 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:17.202254    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb"
Aug 04 04:12:17.319088 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:17.318999    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx"]
Aug 04 04:12:17.320939 ip-10-0-138-99 kubenswrapper[2579]: W0804 04:12:17.320891    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cdb62bc_26d6_42dd_ab6c_93c462d537b2.slice/crio-52d231fd0c8e88302e70a4f4e4c14a060ee1a54ab193e53d12a8e57c37138ac4 WatchSource:0}: Error finding container 52d231fd0c8e88302e70a4f4e4c14a060ee1a54ab193e53d12a8e57c37138ac4: Status 404 returned error can't find the container with id 52d231fd0c8e88302e70a4f4e4c14a060ee1a54ab193e53d12a8e57c37138ac4
Aug 04 04:12:17.588590 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:17.588557    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb"]
Aug 04 04:12:17.591224 ip-10-0-138-99 kubenswrapper[2579]: W0804 04:12:17.591193    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76c67ae_9ab0_4e3f_990d_e5a11a736e21.slice/crio-1cf6fa4176d56fc0600091a0f2a6e8989b34a002cfc42a12682698980fdfad2a WatchSource:0}: Error finding container 1cf6fa4176d56fc0600091a0f2a6e8989b34a002cfc42a12682698980fdfad2a: Status 404 returned error can't find the container with id 1cf6fa4176d56fc0600091a0f2a6e8989b34a002cfc42a12682698980fdfad2a
Aug 04 04:12:18.010657 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:18.010625    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx" event={"ID":"4cdb62bc-26d6-42dd-ab6c-93c462d537b2","Type":"ContainerStarted","Data":"8891468a197a4df0b9dac85f6dfe90bf68b5362f9735c3334b9ff077109cdc6b"}
Aug 04 04:12:18.010857 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:18.010684    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx" event={"ID":"4cdb62bc-26d6-42dd-ab6c-93c462d537b2","Type":"ContainerStarted","Data":"52d231fd0c8e88302e70a4f4e4c14a060ee1a54ab193e53d12a8e57c37138ac4"}
Aug 04 04:12:18.010857 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:18.010711    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx"
Aug 04 04:12:18.012125 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:18.012096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5" event={"ID":"4e57d680-e609-4c20-ac5e-9bb7ae9bf560","Type":"ContainerStarted","Data":"d035be845d1c1e353992679d973f265976c9e15da7c4f6abedab09a81fef85a5"}
Aug 04 04:12:18.012245 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:18.012147    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5" event={"ID":"4e57d680-e609-4c20-ac5e-9bb7ae9bf560","Type":"ContainerStarted","Data":"9d3e5812a065b552381d43e3c62f779be1545525e948b0801403177527084a7f"}
Aug 04 04:12:18.012245 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:18.012168    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"
Aug 04 04:12:18.013506 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:18.013486    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb" event={"ID":"e76c67ae-9ab0-4e3f-990d-e5a11a736e21","Type":"ContainerStarted","Data":"068061a252e1161aebe351e0db010548384f71a95a4685d033978b6a0d8a5129"}
Aug 04 04:12:18.013629 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:18.013519    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb" event={"ID":"e76c67ae-9ab0-4e3f-990d-e5a11a736e21","Type":"ContainerStarted","Data":"1cf6fa4176d56fc0600091a0f2a6e8989b34a002cfc42a12682698980fdfad2a"}
Aug 04 04:12:18.013629 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:18.013583    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb"
Aug 04 04:12:18.029331 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:18.029285    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx" podStartSLOduration=2.029270778 podStartE2EDuration="2.029270778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:12:18.027769271 +0000 UTC m=+3768.707556402" watchObservedRunningTime="2026-08-04 04:12:18.029270778 +0000 UTC m=+3768.709057875"
Aug 04 04:12:18.046548 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:18.046493    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb" podStartSLOduration=2.046479292 podStartE2EDuration="2.046479292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:12:18.045263791 +0000 UTC m=+3768.725050915" watchObservedRunningTime="2026-08-04 04:12:18.046479292 +0000 UTC m=+3768.726266410"
Aug 04 04:12:18.063622 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:18.063564    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5" podStartSLOduration=1.588043155 podStartE2EDuration="2.06354728s" totalImagesPullingTime="475.504125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:12:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:12:18.062279209 +0000 UTC m=+3768.742066329" watchObservedRunningTime="2026-08-04 04:12:18.06354728 +0000 UTC m=+3768.743334479"
Aug 04 04:12:24.024868 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:24.024795    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"
Aug 04 04:12:29.021380 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:29.021348    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx"
Aug 04 04:12:29.021894 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:29.021467    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb"
Aug 04 04:12:45.332498 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:45.332464    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"]
Aug 04 04:12:45.336210 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:45.336192    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"
Aug 04 04:12:45.338946 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:45.338922    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b0089f-serving-cert\""
Aug 04 04:12:45.348242 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:45.348218    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"]
Aug 04 04:12:45.468615 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:45.468568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27bd5e08-1294-4ef1-a392-a6ca80f30d99-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg\" (UID: \"27bd5e08-1294-4ef1-a392-a6ca80f30d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"
Aug 04 04:12:45.468794 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:45.468690    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt4jp\" (UniqueName: \"kubernetes.io/projected/27bd5e08-1294-4ef1-a392-a6ca80f30d99-kube-api-access-bt4jp\") pod \"maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg\" (UID: \"27bd5e08-1294-4ef1-a392-a6ca80f30d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"
Aug 04 04:12:45.468849 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:45.468792    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27bd5e08-1294-4ef1-a392-a6ca80f30d99-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg\" (UID: \"27bd5e08-1294-4ef1-a392-a6ca80f30d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"
Aug 04 04:12:45.570414 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:45.570374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt4jp\" (UniqueName: \"kubernetes.io/projected/27bd5e08-1294-4ef1-a392-a6ca80f30d99-kube-api-access-bt4jp\") pod \"maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg\" (UID: \"27bd5e08-1294-4ef1-a392-a6ca80f30d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"
Aug 04 04:12:45.570614 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:45.570484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27bd5e08-1294-4ef1-a392-a6ca80f30d99-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg\" (UID: \"27bd5e08-1294-4ef1-a392-a6ca80f30d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"
Aug 04 04:12:45.570614 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:45.570547    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27bd5e08-1294-4ef1-a392-a6ca80f30d99-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg\" (UID: \"27bd5e08-1294-4ef1-a392-a6ca80f30d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"
Aug 04 04:12:45.570730 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:12:45.570639    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-serving-cert: secret "maas-api-e2e-ipp-b-b0089f-serving-cert" not found
Aug 04 04:12:45.570730 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:12:45.570709    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27bd5e08-1294-4ef1-a392-a6ca80f30d99-maas-api-tls podName:27bd5e08-1294-4ef1-a392-a6ca80f30d99 nodeName:}" failed. No retries permitted until 2026-08-04 04:12:46.070690938 +0000 UTC m=+3796.750478042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/27bd5e08-1294-4ef1-a392-a6ca80f30d99-maas-api-tls") pod "maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg" (UID: "27bd5e08-1294-4ef1-a392-a6ca80f30d99") : secret "maas-api-e2e-ipp-b-b0089f-serving-cert" not found
Aug 04 04:12:45.571192 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:45.571172    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27bd5e08-1294-4ef1-a392-a6ca80f30d99-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg\" (UID: \"27bd5e08-1294-4ef1-a392-a6ca80f30d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"
Aug 04 04:12:45.579462 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:45.579438    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt4jp\" (UniqueName: \"kubernetes.io/projected/27bd5e08-1294-4ef1-a392-a6ca80f30d99-kube-api-access-bt4jp\") pod \"maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg\" (UID: \"27bd5e08-1294-4ef1-a392-a6ca80f30d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"
Aug 04 04:12:45.967398 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:45.967356    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g"]
Aug 04 04:12:45.971393 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:45.971370    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g"
Aug 04 04:12:45.974316 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:45.974296    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b0089f\""
Aug 04 04:12:45.974502 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:45.974486    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b0089f-dockercfg-4ss2r\""
Aug 04 04:12:45.983442 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:45.983416    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g"]
Aug 04 04:12:46.022729 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.022693    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn"]
Aug 04 04:12:46.026528 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.026509    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn"
Aug 04 04:12:46.042047 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.042020    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn"]
Aug 04 04:12:46.078196 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.078167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27bd5e08-1294-4ef1-a392-a6ca80f30d99-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg\" (UID: \"27bd5e08-1294-4ef1-a392-a6ca80f30d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"
Aug 04 04:12:46.078400 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.078245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g\" (UID: \"04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g"
Aug 04 04:12:46.078400 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.078317    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8gj4\" (UniqueName: \"kubernetes.io/projected/04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19-kube-api-access-q8gj4\") pod \"payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g\" (UID: \"04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g"
Aug 04 04:12:46.080811 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.080786    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27bd5e08-1294-4ef1-a392-a6ca80f30d99-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg\" (UID: \"27bd5e08-1294-4ef1-a392-a6ca80f30d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"
Aug 04 04:12:46.178908 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.178878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grwj5\" (UniqueName: \"kubernetes.io/projected/fdd912fe-891f-4b8b-994c-fd8baa7c6f87-kube-api-access-grwj5\") pod \"payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn\" (UID: \"fdd912fe-891f-4b8b-994c-fd8baa7c6f87\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn"
Aug 04 04:12:46.179114 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.178918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd912fe-891f-4b8b-994c-fd8baa7c6f87-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn\" (UID: \"fdd912fe-891f-4b8b-994c-fd8baa7c6f87\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn"
Aug 04 04:12:46.179114 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.178949    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g\" (UID: \"04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g"
Aug 04 04:12:46.179114 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.179059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8gj4\" (UniqueName: \"kubernetes.io/projected/04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19-kube-api-access-q8gj4\") pod \"payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g\" (UID: \"04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g"
Aug 04 04:12:46.179714 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.179693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g\" (UID: \"04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g"
Aug 04 04:12:46.188035 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.188010    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8gj4\" (UniqueName: \"kubernetes.io/projected/04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19-kube-api-access-q8gj4\") pod \"payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g\" (UID: \"04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g"
Aug 04 04:12:46.250094 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.250007    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"
Aug 04 04:12:46.279947 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.279917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grwj5\" (UniqueName: \"kubernetes.io/projected/fdd912fe-891f-4b8b-994c-fd8baa7c6f87-kube-api-access-grwj5\") pod \"payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn\" (UID: \"fdd912fe-891f-4b8b-994c-fd8baa7c6f87\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn"
Aug 04 04:12:46.280100 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.279994    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd912fe-891f-4b8b-994c-fd8baa7c6f87-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn\" (UID: \"fdd912fe-891f-4b8b-994c-fd8baa7c6f87\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn"
Aug 04 04:12:46.280738 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.280713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd912fe-891f-4b8b-994c-fd8baa7c6f87-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn\" (UID: \"fdd912fe-891f-4b8b-994c-fd8baa7c6f87\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn"
Aug 04 04:12:46.282703 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.282678    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g"
Aug 04 04:12:46.291398 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.291368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grwj5\" (UniqueName: \"kubernetes.io/projected/fdd912fe-891f-4b8b-994c-fd8baa7c6f87-kube-api-access-grwj5\") pod \"payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn\" (UID: \"fdd912fe-891f-4b8b-994c-fd8baa7c6f87\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn"
Aug 04 04:12:46.336422 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.336393    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn"
Aug 04 04:12:46.414883 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.413416    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"]
Aug 04 04:12:46.417955 ip-10-0-138-99 kubenswrapper[2579]: W0804 04:12:46.417842    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27bd5e08_1294_4ef1_a392_a6ca80f30d99.slice/crio-0da0e69a67c9f595e699b385b9ccee1a1e7ed83ad7df8feee339f57d0fb156c4 WatchSource:0}: Error finding container 0da0e69a67c9f595e699b385b9ccee1a1e7ed83ad7df8feee339f57d0fb156c4: Status 404 returned error can't find the container with id 0da0e69a67c9f595e699b385b9ccee1a1e7ed83ad7df8feee339f57d0fb156c4
Aug 04 04:12:46.478873 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.478847    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g"]
Aug 04 04:12:46.482760 ip-10-0-138-99 kubenswrapper[2579]: W0804 04:12:46.482728    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cdf67b_8c92_4ddd_bbfc_3ceeffd75b19.slice/crio-098431aac776ec1e46972684aa76ba6f1fe6abf613ec5cad55a467bf4a370354 WatchSource:0}: Error finding container 098431aac776ec1e46972684aa76ba6f1fe6abf613ec5cad55a467bf4a370354: Status 404 returned error can't find the container with id 098431aac776ec1e46972684aa76ba6f1fe6abf613ec5cad55a467bf4a370354
Aug 04 04:12:46.715368 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:46.715346    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn"]
Aug 04 04:12:46.717074 ip-10-0-138-99 kubenswrapper[2579]: W0804 04:12:46.717047    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd912fe_891f_4b8b_994c_fd8baa7c6f87.slice/crio-3620dddb202d70a55967a3f395d85e6571f87d549344bf76da6a788b02db624c WatchSource:0}: Error finding container 3620dddb202d70a55967a3f395d85e6571f87d549344bf76da6a788b02db624c: Status 404 returned error can't find the container with id 3620dddb202d70a55967a3f395d85e6571f87d549344bf76da6a788b02db624c
Aug 04 04:12:47.180742 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:47.180637    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg" event={"ID":"27bd5e08-1294-4ef1-a392-a6ca80f30d99","Type":"ContainerStarted","Data":"a8a189f0b870e200546f1b5725edfaa85dee8539cbc1cc4f919118301e86f4aa"}
Aug 04 04:12:47.180742 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:47.180705    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg" event={"ID":"27bd5e08-1294-4ef1-a392-a6ca80f30d99","Type":"ContainerStarted","Data":"0da0e69a67c9f595e699b385b9ccee1a1e7ed83ad7df8feee339f57d0fb156c4"}
Aug 04 04:12:47.180742 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:47.180761    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"
Aug 04 04:12:47.182704 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:47.182676    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn" event={"ID":"fdd912fe-891f-4b8b-994c-fd8baa7c6f87","Type":"ContainerStarted","Data":"650a035ace15b60e838b10f4950509027e161e3d83e58c0c8b0a663c8227f3ae"}
Aug 04 04:12:47.182843 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:47.182727    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn" event={"ID":"fdd912fe-891f-4b8b-994c-fd8baa7c6f87","Type":"ContainerStarted","Data":"3620dddb202d70a55967a3f395d85e6571f87d549344bf76da6a788b02db624c"}
Aug 04 04:12:47.182843 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:47.182770    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn"
Aug 04 04:12:47.193039 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:47.193000    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g" event={"ID":"04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19","Type":"ContainerStarted","Data":"d69e7048da7ffa97dafcdf8f3f815f8ab5accd0ccbfebcf9c9dd0c543fe482b4"}
Aug 04 04:12:47.193204 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:47.193191    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g"
Aug 04 04:12:47.193247 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:47.193215    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g" event={"ID":"04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19","Type":"ContainerStarted","Data":"098431aac776ec1e46972684aa76ba6f1fe6abf613ec5cad55a467bf4a370354"}
Aug 04 04:12:47.199927 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:47.199878    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg" podStartSLOduration=1.749400898 podStartE2EDuration="2.199863255s" totalImagesPullingTime="450.462357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:12:47.198728218 +0000 UTC m=+3797.878515356" watchObservedRunningTime="2026-08-04 04:12:47.199863255 +0000 UTC m=+3797.879650374"
Aug 04 04:12:47.216798 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:47.216751    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g" podStartSLOduration=2.216732172 podStartE2EDuration="2.216732172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:12:47.215476735 +0000 UTC m=+3797.895263883" watchObservedRunningTime="2026-08-04 04:12:47.216732172 +0000 UTC m=+3797.896519335"
Aug 04 04:12:47.234344 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:47.234291    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn" podStartSLOduration=1.234276774 podStartE2EDuration="1.234276774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:12:47.232771649 +0000 UTC m=+3797.912558785" watchObservedRunningTime="2026-08-04 04:12:47.234276774 +0000 UTC m=+3797.914063940"
Aug 04 04:12:53.205102 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:53.205067    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"
Aug 04 04:12:58.201580 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:58.201551    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn"
Aug 04 04:12:58.202100 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:12:58.201629    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g"
Aug 04 04:13:16.050146 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:13:16.050111    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-46wnj_2dbe90ed-02ed-428f-ad98-ae60193a9e66/payload-processing/0.log"
Aug 04 04:13:16.164432 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:13:16.164396    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g_04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19/payload-processing/0.log"
Aug 04 04:14:06.659537 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:06.659509    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx_4cdb62bc-26d6-42dd-ab6c-93c462d537b2/payload-processing/0.log"
Aug 04 04:14:06.798946 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:06.798907    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-46wnj_2dbe90ed-02ed-428f-ad98-ae60193a9e66/payload-processing/0.log"
Aug 04 04:14:06.949518 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:06.949438    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g_04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19/payload-processing/0.log"
Aug 04 04:14:14.887720 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:14.887679    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"]
Aug 04 04:14:14.891647 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:14.891624    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"
Aug 04 04:14:14.894303 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:14.894281    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-feaa4d-serving-cert\""
Aug 04 04:14:14.901097 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:14.901075    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"]
Aug 04 04:14:14.954751 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:14.954715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/243466d7-23f2-4e47-9660-ab21ab9b8eed-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9\" (UID: \"243466d7-23f2-4e47-9660-ab21ab9b8eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"
Aug 04 04:14:14.954931 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:14.954768    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skhck\" (UniqueName: \"kubernetes.io/projected/243466d7-23f2-4e47-9660-ab21ab9b8eed-kube-api-access-skhck\") pod \"maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9\" (UID: \"243466d7-23f2-4e47-9660-ab21ab9b8eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"
Aug 04 04:14:14.954931 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:14.954788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/243466d7-23f2-4e47-9660-ab21ab9b8eed-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9\" (UID: \"243466d7-23f2-4e47-9660-ab21ab9b8eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"
Aug 04 04:14:15.055749 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.055713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/243466d7-23f2-4e47-9660-ab21ab9b8eed-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9\" (UID: \"243466d7-23f2-4e47-9660-ab21ab9b8eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"
Aug 04 04:14:15.055942 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.055777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skhck\" (UniqueName: \"kubernetes.io/projected/243466d7-23f2-4e47-9660-ab21ab9b8eed-kube-api-access-skhck\") pod \"maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9\" (UID: \"243466d7-23f2-4e47-9660-ab21ab9b8eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"
Aug 04 04:14:15.055942 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.055801    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/243466d7-23f2-4e47-9660-ab21ab9b8eed-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9\" (UID: \"243466d7-23f2-4e47-9660-ab21ab9b8eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"
Aug 04 04:14:15.055942 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:14:15.055932    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-serving-cert: secret "maas-api-e2e-ipp-cleanup-feaa4d-serving-cert" not found
Aug 04 04:14:15.056097 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:14:15.056014    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243466d7-23f2-4e47-9660-ab21ab9b8eed-maas-api-tls podName:243466d7-23f2-4e47-9660-ab21ab9b8eed nodeName:}" failed. No retries permitted until 2026-08-04 04:14:15.555995417 +0000 UTC m=+3886.235782525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/243466d7-23f2-4e47-9660-ab21ab9b8eed-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9" (UID: "243466d7-23f2-4e47-9660-ab21ab9b8eed") : secret "maas-api-e2e-ipp-cleanup-feaa4d-serving-cert" not found
Aug 04 04:14:15.056567 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.056546    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/243466d7-23f2-4e47-9660-ab21ab9b8eed-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9\" (UID: \"243466d7-23f2-4e47-9660-ab21ab9b8eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"
Aug 04 04:14:15.066501 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.066478    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skhck\" (UniqueName: \"kubernetes.io/projected/243466d7-23f2-4e47-9660-ab21ab9b8eed-kube-api-access-skhck\") pod \"maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9\" (UID: \"243466d7-23f2-4e47-9660-ab21ab9b8eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"
Aug 04 04:14:15.560321 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.560288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/243466d7-23f2-4e47-9660-ab21ab9b8eed-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9\" (UID: \"243466d7-23f2-4e47-9660-ab21ab9b8eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"
Aug 04 04:14:15.560544 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.560520    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc"]
Aug 04 04:14:15.562772 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.562750    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/243466d7-23f2-4e47-9660-ab21ab9b8eed-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9\" (UID: \"243466d7-23f2-4e47-9660-ab21ab9b8eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"
Aug 04 04:14:15.564485 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.564468    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc"
Aug 04 04:14:15.567523 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.567502    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-feaa4d-dockercfg-vm9rj\""
Aug 04 04:14:15.567779 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.567759    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-feaa4d\""
Aug 04 04:14:15.577274 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.577250    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc"]
Aug 04 04:14:15.661480 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.661446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2j92\" (UniqueName: \"kubernetes.io/projected/02852acd-9031-4aee-adb3-85396e768bf9-kube-api-access-s2j92\") pod \"payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc\" (UID: \"02852acd-9031-4aee-adb3-85396e768bf9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc"
Aug 04 04:14:15.661693 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.661583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02852acd-9031-4aee-adb3-85396e768bf9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc\" (UID: \"02852acd-9031-4aee-adb3-85396e768bf9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc"
Aug 04 04:14:15.762116 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.762078    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02852acd-9031-4aee-adb3-85396e768bf9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc\" (UID: \"02852acd-9031-4aee-adb3-85396e768bf9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc"
Aug 04 04:14:15.762297 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.762175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2j92\" (UniqueName: \"kubernetes.io/projected/02852acd-9031-4aee-adb3-85396e768bf9-kube-api-access-s2j92\") pod \"payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc\" (UID: \"02852acd-9031-4aee-adb3-85396e768bf9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc"
Aug 04 04:14:15.762811 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.762790    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02852acd-9031-4aee-adb3-85396e768bf9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc\" (UID: \"02852acd-9031-4aee-adb3-85396e768bf9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc"
Aug 04 04:14:15.773213 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.773179    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2j92\" (UniqueName: \"kubernetes.io/projected/02852acd-9031-4aee-adb3-85396e768bf9-kube-api-access-s2j92\") pod \"payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc\" (UID: \"02852acd-9031-4aee-adb3-85396e768bf9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc"
Aug 04 04:14:15.807430 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.807397    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"
Aug 04 04:14:15.884286 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.884208    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc"
Aug 04 04:14:15.964876 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:15.964841    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"]
Aug 04 04:14:15.967157 ip-10-0-138-99 kubenswrapper[2579]: W0804 04:14:15.967130    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243466d7_23f2_4e47_9660_ab21ab9b8eed.slice/crio-eac30c2cd70ba057a851fe4c44e7a0e97a9a14c9e955afadebc1ee7606059970 WatchSource:0}: Error finding container eac30c2cd70ba057a851fe4c44e7a0e97a9a14c9e955afadebc1ee7606059970: Status 404 returned error can't find the container with id eac30c2cd70ba057a851fe4c44e7a0e97a9a14c9e955afadebc1ee7606059970
Aug 04 04:14:16.042915 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:16.042888    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc"]
Aug 04 04:14:16.045372 ip-10-0-138-99 kubenswrapper[2579]: W0804 04:14:16.045342    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02852acd_9031_4aee_adb3_85396e768bf9.slice/crio-d0879f14c15783bf78c69f34d4101267ce61f8d23d846f6c2bd1984d4ea04793 WatchSource:0}: Error finding container d0879f14c15783bf78c69f34d4101267ce61f8d23d846f6c2bd1984d4ea04793: Status 404 returned error can't find the container with id d0879f14c15783bf78c69f34d4101267ce61f8d23d846f6c2bd1984d4ea04793
Aug 04 04:14:16.694371 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:16.694264    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9" event={"ID":"243466d7-23f2-4e47-9660-ab21ab9b8eed","Type":"ContainerStarted","Data":"4d92b2cfd5b5a252fc5120dc2cbe1b4aa48276e867e1e7673adabc08c748cda9"}
Aug 04 04:14:16.694371 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:16.694324    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9" event={"ID":"243466d7-23f2-4e47-9660-ab21ab9b8eed","Type":"ContainerStarted","Data":"eac30c2cd70ba057a851fe4c44e7a0e97a9a14c9e955afadebc1ee7606059970"}
Aug 04 04:14:16.694371 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:16.694351    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"
Aug 04 04:14:16.695911 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:16.695877    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc" event={"ID":"02852acd-9031-4aee-adb3-85396e768bf9","Type":"ContainerStarted","Data":"71ba7db33fe9fbc565fb1d32f3da66fa3cf88dde296299b0239c44bee178d722"}
Aug 04 04:14:16.696050 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:16.695918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc" event={"ID":"02852acd-9031-4aee-adb3-85396e768bf9","Type":"ContainerStarted","Data":"d0879f14c15783bf78c69f34d4101267ce61f8d23d846f6c2bd1984d4ea04793"}
Aug 04 04:14:16.696050 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:16.696002    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc"
Aug 04 04:14:16.711875 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:16.711828    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9" podStartSLOduration=2.319524226 podStartE2EDuration="2.711811673s" totalImagesPullingTime="392.287447ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:14:16.71023165 +0000 UTC m=+3887.390018774" watchObservedRunningTime="2026-08-04 04:14:16.711811673 +0000 UTC m=+3887.391598792"
Aug 04 04:14:16.749687 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:16.749626    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc" podStartSLOduration=1.7496013289999999 podStartE2EDuration="1.749601329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:14:16.746931867 +0000 UTC m=+3887.426718987" watchObservedRunningTime="2026-08-04 04:14:16.749601329 +0000 UTC m=+3887.429388451"
Aug 04 04:14:22.707938 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:22.707908    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"
Aug 04 04:14:27.704157 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:27.704124    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc"
Aug 04 04:14:45.526055 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.526010    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"]
Aug 04 04:14:45.526654 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.526357    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9" podUID="243466d7-23f2-4e47-9660-ab21ab9b8eed" containerName="maas-api" containerID="cri-o://4d92b2cfd5b5a252fc5120dc2cbe1b4aa48276e867e1e7673adabc08c748cda9" gracePeriod=30
Aug 04 04:14:45.588607 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.588574    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc"]
Aug 04 04:14:45.589039 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.588980    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc" podUID="02852acd-9031-4aee-adb3-85396e768bf9" containerName="payload-processing" containerID="cri-o://71ba7db33fe9fbc565fb1d32f3da66fa3cf88dde296299b0239c44bee178d722" gracePeriod=30
Aug 04 04:14:45.859314 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.859276    2579 generic.go:356] "Generic (PLEG): container finished" podID="02852acd-9031-4aee-adb3-85396e768bf9" containerID="71ba7db33fe9fbc565fb1d32f3da66fa3cf88dde296299b0239c44bee178d722" exitCode=0
Aug 04 04:14:45.859456 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.859348    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc" event={"ID":"02852acd-9031-4aee-adb3-85396e768bf9","Type":"ContainerDied","Data":"71ba7db33fe9fbc565fb1d32f3da66fa3cf88dde296299b0239c44bee178d722"}
Aug 04 04:14:45.860913 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.860890    2579 generic.go:356] "Generic (PLEG): container finished" podID="243466d7-23f2-4e47-9660-ab21ab9b8eed" containerID="4d92b2cfd5b5a252fc5120dc2cbe1b4aa48276e867e1e7673adabc08c748cda9" exitCode=0
Aug 04 04:14:45.861086 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.860988    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9" event={"ID":"243466d7-23f2-4e47-9660-ab21ab9b8eed","Type":"ContainerDied","Data":"4d92b2cfd5b5a252fc5120dc2cbe1b4aa48276e867e1e7673adabc08c748cda9"}
Aug 04 04:14:45.861086 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.861037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9" event={"ID":"243466d7-23f2-4e47-9660-ab21ab9b8eed","Type":"ContainerDied","Data":"eac30c2cd70ba057a851fe4c44e7a0e97a9a14c9e955afadebc1ee7606059970"}
Aug 04 04:14:45.861086 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.861060    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eac30c2cd70ba057a851fe4c44e7a0e97a9a14c9e955afadebc1ee7606059970"
Aug 04 04:14:45.863557 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.863535    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"
Aug 04 04:14:45.881055 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.881029    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc"
Aug 04 04:14:45.936151 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.936123    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/243466d7-23f2-4e47-9660-ab21ab9b8eed-maas-api-tls\") pod \"243466d7-23f2-4e47-9660-ab21ab9b8eed\" (UID: \"243466d7-23f2-4e47-9660-ab21ab9b8eed\") "
Aug 04 04:14:45.936329 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.936212    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/243466d7-23f2-4e47-9660-ab21ab9b8eed-openshift-service-ca\") pod \"243466d7-23f2-4e47-9660-ab21ab9b8eed\" (UID: \"243466d7-23f2-4e47-9660-ab21ab9b8eed\") "
Aug 04 04:14:45.936329 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.936231    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skhck\" (UniqueName: \"kubernetes.io/projected/243466d7-23f2-4e47-9660-ab21ab9b8eed-kube-api-access-skhck\") pod \"243466d7-23f2-4e47-9660-ab21ab9b8eed\" (UID: \"243466d7-23f2-4e47-9660-ab21ab9b8eed\") "
Aug 04 04:14:45.936329 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.936297    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02852acd-9031-4aee-adb3-85396e768bf9-plugins-config-volume\") pod \"02852acd-9031-4aee-adb3-85396e768bf9\" (UID: \"02852acd-9031-4aee-adb3-85396e768bf9\") "
Aug 04 04:14:45.936433 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.936340    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2j92\" (UniqueName: \"kubernetes.io/projected/02852acd-9031-4aee-adb3-85396e768bf9-kube-api-access-s2j92\") pod \"02852acd-9031-4aee-adb3-85396e768bf9\" (UID: \"02852acd-9031-4aee-adb3-85396e768bf9\") "
Aug 04 04:14:45.936610 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.936586    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/243466d7-23f2-4e47-9660-ab21ab9b8eed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "243466d7-23f2-4e47-9660-ab21ab9b8eed" (UID: "243466d7-23f2-4e47-9660-ab21ab9b8eed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 04:14:45.936724 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.936696    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02852acd-9031-4aee-adb3-85396e768bf9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "02852acd-9031-4aee-adb3-85396e768bf9" (UID: "02852acd-9031-4aee-adb3-85396e768bf9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 04:14:45.938392 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.938362    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02852acd-9031-4aee-adb3-85396e768bf9-kube-api-access-s2j92" (OuterVolumeSpecName: "kube-api-access-s2j92") pod "02852acd-9031-4aee-adb3-85396e768bf9" (UID: "02852acd-9031-4aee-adb3-85396e768bf9"). InnerVolumeSpecName "kube-api-access-s2j92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:14:45.938496 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.938396    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243466d7-23f2-4e47-9660-ab21ab9b8eed-kube-api-access-skhck" (OuterVolumeSpecName: "kube-api-access-skhck") pod "243466d7-23f2-4e47-9660-ab21ab9b8eed" (UID: "243466d7-23f2-4e47-9660-ab21ab9b8eed"). InnerVolumeSpecName "kube-api-access-skhck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:14:45.938496 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:45.938464    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/243466d7-23f2-4e47-9660-ab21ab9b8eed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "243466d7-23f2-4e47-9660-ab21ab9b8eed" (UID: "243466d7-23f2-4e47-9660-ab21ab9b8eed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 04:14:46.037405 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:46.037369    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2j92\" (UniqueName: \"kubernetes.io/projected/02852acd-9031-4aee-adb3-85396e768bf9-kube-api-access-s2j92\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:14:46.037405 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:46.037399    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/243466d7-23f2-4e47-9660-ab21ab9b8eed-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:14:46.037405 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:46.037409    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/243466d7-23f2-4e47-9660-ab21ab9b8eed-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:14:46.037653 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:46.037420    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skhck\" (UniqueName: \"kubernetes.io/projected/243466d7-23f2-4e47-9660-ab21ab9b8eed-kube-api-access-skhck\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:14:46.037653 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:46.037429    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02852acd-9031-4aee-adb3-85396e768bf9-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:14:46.868411 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:46.868323    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc"
Aug 04 04:14:46.868411 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:46.868353    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc" event={"ID":"02852acd-9031-4aee-adb3-85396e768bf9","Type":"ContainerDied","Data":"d0879f14c15783bf78c69f34d4101267ce61f8d23d846f6c2bd1984d4ea04793"}
Aug 04 04:14:46.868917 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:46.868433    2579 scope.go:117] "RemoveContainer" containerID="71ba7db33fe9fbc565fb1d32f3da66fa3cf88dde296299b0239c44bee178d722"
Aug 04 04:14:46.868917 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:46.868509    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"
Aug 04 04:14:46.894070 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:46.894039    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc"]
Aug 04 04:14:46.898063 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:46.898042    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-feaa4d-868995c59d-lmgcc"]
Aug 04 04:14:46.911531 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:46.911510    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"]
Aug 04 04:14:46.915409 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:46.915384    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-feaa4d-799d5b76c4-44cn9"]
Aug 04 04:14:47.846302 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:47.846266    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02852acd-9031-4aee-adb3-85396e768bf9" path="/var/lib/kubelet/pods/02852acd-9031-4aee-adb3-85396e768bf9/volumes"
Aug 04 04:14:47.846784 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:14:47.846762    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="243466d7-23f2-4e47-9660-ab21ab9b8eed" path="/var/lib/kubelet/pods/243466d7-23f2-4e47-9660-ab21ab9b8eed/volumes"
Aug 04 04:15:11.634334 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:11.634286    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"]
Aug 04 04:15:11.634796 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:11.634710    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5" podUID="4e57d680-e609-4c20-ac5e-9bb7ae9bf560" containerName="maas-api" containerID="cri-o://d035be845d1c1e353992679d973f265976c9e15da7c4f6abedab09a81fef85a5" gracePeriod=30
Aug 04 04:15:11.726331 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:11.726293    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx"]
Aug 04 04:15:11.726556 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:11.726525    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx" podUID="4cdb62bc-26d6-42dd-ab6c-93c462d537b2" containerName="payload-processing" containerID="cri-o://8891468a197a4df0b9dac85f6dfe90bf68b5362f9735c3334b9ff077109cdc6b" gracePeriod=30
Aug 04 04:15:11.753607 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:11.753580    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb"]
Aug 04 04:15:11.753831 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:11.753806    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb" podUID="e76c67ae-9ab0-4e3f-990d-e5a11a736e21" containerName="payload-pre-processing" containerID="cri-o://068061a252e1161aebe351e0db010548384f71a95a4685d033978b6a0d8a5129" gracePeriod=30
Aug 04 04:15:11.993101 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:11.992777    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"
Aug 04 04:15:11.996162 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:11.996139    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-maas-api-tls\") pod \"4e57d680-e609-4c20-ac5e-9bb7ae9bf560\" (UID: \"4e57d680-e609-4c20-ac5e-9bb7ae9bf560\") "
Aug 04 04:15:11.998336 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:11.998303    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4e57d680-e609-4c20-ac5e-9bb7ae9bf560" (UID: "4e57d680-e609-4c20-ac5e-9bb7ae9bf560"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 04:15:12.008604 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.008574    2579 generic.go:356] "Generic (PLEG): container finished" podID="4cdb62bc-26d6-42dd-ab6c-93c462d537b2" containerID="8891468a197a4df0b9dac85f6dfe90bf68b5362f9735c3334b9ff077109cdc6b" exitCode=0
Aug 04 04:15:12.008754 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.008652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx" event={"ID":"4cdb62bc-26d6-42dd-ab6c-93c462d537b2","Type":"ContainerDied","Data":"8891468a197a4df0b9dac85f6dfe90bf68b5362f9735c3334b9ff077109cdc6b"}
Aug 04 04:15:12.012229 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.011947    2579 generic.go:356] "Generic (PLEG): container finished" podID="e76c67ae-9ab0-4e3f-990d-e5a11a736e21" containerID="068061a252e1161aebe351e0db010548384f71a95a4685d033978b6a0d8a5129" exitCode=0
Aug 04 04:15:12.012229 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.012155    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb" event={"ID":"e76c67ae-9ab0-4e3f-990d-e5a11a736e21","Type":"ContainerDied","Data":"068061a252e1161aebe351e0db010548384f71a95a4685d033978b6a0d8a5129"}
Aug 04 04:15:12.014680 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.013801    2579 generic.go:356] "Generic (PLEG): container finished" podID="4e57d680-e609-4c20-ac5e-9bb7ae9bf560" containerID="d035be845d1c1e353992679d973f265976c9e15da7c4f6abedab09a81fef85a5" exitCode=0
Aug 04 04:15:12.014680 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.013857    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5" event={"ID":"4e57d680-e609-4c20-ac5e-9bb7ae9bf560","Type":"ContainerDied","Data":"d035be845d1c1e353992679d973f265976c9e15da7c4f6abedab09a81fef85a5"}
Aug 04 04:15:12.014680 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.013897    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5" event={"ID":"4e57d680-e609-4c20-ac5e-9bb7ae9bf560","Type":"ContainerDied","Data":"9d3e5812a065b552381d43e3c62f779be1545525e948b0801403177527084a7f"}
Aug 04 04:15:12.014680 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.013927    2579 scope.go:117] "RemoveContainer" containerID="d035be845d1c1e353992679d973f265976c9e15da7c4f6abedab09a81fef85a5"
Aug 04 04:15:12.014680 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.013951    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"
Aug 04 04:15:12.025692 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.025655    2579 scope.go:117] "RemoveContainer" containerID="d035be845d1c1e353992679d973f265976c9e15da7c4f6abedab09a81fef85a5"
Aug 04 04:15:12.026069 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:15:12.025989    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d035be845d1c1e353992679d973f265976c9e15da7c4f6abedab09a81fef85a5\": container with ID starting with d035be845d1c1e353992679d973f265976c9e15da7c4f6abedab09a81fef85a5 not found: ID does not exist" containerID="d035be845d1c1e353992679d973f265976c9e15da7c4f6abedab09a81fef85a5"
Aug 04 04:15:12.026069 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.026030    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d035be845d1c1e353992679d973f265976c9e15da7c4f6abedab09a81fef85a5"} err="failed to get container status \"d035be845d1c1e353992679d973f265976c9e15da7c4f6abedab09a81fef85a5\": rpc error: code = NotFound desc = could not find container \"d035be845d1c1e353992679d973f265976c9e15da7c4f6abedab09a81fef85a5\": container with ID starting with d035be845d1c1e353992679d973f265976c9e15da7c4f6abedab09a81fef85a5 not found: ID does not exist"
Aug 04 04:15:12.053261 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.053239    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx"
Aug 04 04:15:12.060353 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.060331    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb"
Aug 04 04:15:12.097412 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.097380    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cdb62bc-26d6-42dd-ab6c-93c462d537b2-plugins-config-volume\") pod \"4cdb62bc-26d6-42dd-ab6c-93c462d537b2\" (UID: \"4cdb62bc-26d6-42dd-ab6c-93c462d537b2\") "
Aug 04 04:15:12.097584 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.097426    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvj75\" (UniqueName: \"kubernetes.io/projected/e76c67ae-9ab0-4e3f-990d-e5a11a736e21-kube-api-access-rvj75\") pod \"e76c67ae-9ab0-4e3f-990d-e5a11a736e21\" (UID: \"e76c67ae-9ab0-4e3f-990d-e5a11a736e21\") "
Aug 04 04:15:12.097584 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.097457    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j8bp\" (UniqueName: \"kubernetes.io/projected/4cdb62bc-26d6-42dd-ab6c-93c462d537b2-kube-api-access-6j8bp\") pod \"4cdb62bc-26d6-42dd-ab6c-93c462d537b2\" (UID: \"4cdb62bc-26d6-42dd-ab6c-93c462d537b2\") "
Aug 04 04:15:12.097584 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.097536    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hf27\" (UniqueName: \"kubernetes.io/projected/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-kube-api-access-6hf27\") pod \"4e57d680-e609-4c20-ac5e-9bb7ae9bf560\" (UID: \"4e57d680-e609-4c20-ac5e-9bb7ae9bf560\") "
Aug 04 04:15:12.097584 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.097566    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76c67ae-9ab0-4e3f-990d-e5a11a736e21-plugins-config-volume\") pod \"e76c67ae-9ab0-4e3f-990d-e5a11a736e21\" (UID: \"e76c67ae-9ab0-4e3f-990d-e5a11a736e21\") "
Aug 04 04:15:12.097836 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.097604    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-openshift-service-ca\") pod \"4e57d680-e609-4c20-ac5e-9bb7ae9bf560\" (UID: \"4e57d680-e609-4c20-ac5e-9bb7ae9bf560\") "
Aug 04 04:15:12.097836 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.097785    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cdb62bc-26d6-42dd-ab6c-93c462d537b2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4cdb62bc-26d6-42dd-ab6c-93c462d537b2" (UID: "4cdb62bc-26d6-42dd-ab6c-93c462d537b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 04:15:12.098154 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.098111    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cdb62bc-26d6-42dd-ab6c-93c462d537b2-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:15:12.098154 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.098137    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:15:12.098284 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.098160    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4e57d680-e609-4c20-ac5e-9bb7ae9bf560" (UID: "4e57d680-e609-4c20-ac5e-9bb7ae9bf560"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 04:15:12.098284 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.098217    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76c67ae-9ab0-4e3f-990d-e5a11a736e21-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e76c67ae-9ab0-4e3f-990d-e5a11a736e21" (UID: "e76c67ae-9ab0-4e3f-990d-e5a11a736e21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 04:15:12.100187 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.100157    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cdb62bc-26d6-42dd-ab6c-93c462d537b2-kube-api-access-6j8bp" (OuterVolumeSpecName: "kube-api-access-6j8bp") pod "4cdb62bc-26d6-42dd-ab6c-93c462d537b2" (UID: "4cdb62bc-26d6-42dd-ab6c-93c462d537b2"). InnerVolumeSpecName "kube-api-access-6j8bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:15:12.100187 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.100169    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76c67ae-9ab0-4e3f-990d-e5a11a736e21-kube-api-access-rvj75" (OuterVolumeSpecName: "kube-api-access-rvj75") pod "e76c67ae-9ab0-4e3f-990d-e5a11a736e21" (UID: "e76c67ae-9ab0-4e3f-990d-e5a11a736e21"). InnerVolumeSpecName "kube-api-access-rvj75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:15:12.100303 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.100226    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-kube-api-access-6hf27" (OuterVolumeSpecName: "kube-api-access-6hf27") pod "4e57d680-e609-4c20-ac5e-9bb7ae9bf560" (UID: "4e57d680-e609-4c20-ac5e-9bb7ae9bf560"). InnerVolumeSpecName "kube-api-access-6hf27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:15:12.198891 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.198803    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvj75\" (UniqueName: \"kubernetes.io/projected/e76c67ae-9ab0-4e3f-990d-e5a11a736e21-kube-api-access-rvj75\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:15:12.198891 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.198834    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j8bp\" (UniqueName: \"kubernetes.io/projected/4cdb62bc-26d6-42dd-ab6c-93c462d537b2-kube-api-access-6j8bp\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:15:12.198891 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.198844    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hf27\" (UniqueName: \"kubernetes.io/projected/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-kube-api-access-6hf27\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:15:12.198891 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.198854    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76c67ae-9ab0-4e3f-990d-e5a11a736e21-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:15:12.198891 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.198862    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e57d680-e609-4c20-ac5e-9bb7ae9bf560-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:15:12.342339 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.342293    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"]
Aug 04 04:15:12.350492 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:12.350463    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d8d0c-864c54b84c-nl4w5"]
Aug 04 04:15:13.020848 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:13.020817    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx"
Aug 04 04:15:13.021347 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:13.020850    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx" event={"ID":"4cdb62bc-26d6-42dd-ab6c-93c462d537b2","Type":"ContainerDied","Data":"52d231fd0c8e88302e70a4f4e4c14a060ee1a54ab193e53d12a8e57c37138ac4"}
Aug 04 04:15:13.021347 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:13.020911    2579 scope.go:117] "RemoveContainer" containerID="8891468a197a4df0b9dac85f6dfe90bf68b5362f9735c3334b9ff077109cdc6b"
Aug 04 04:15:13.022220 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:13.022066    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb" event={"ID":"e76c67ae-9ab0-4e3f-990d-e5a11a736e21","Type":"ContainerDied","Data":"1cf6fa4176d56fc0600091a0f2a6e8989b34a002cfc42a12682698980fdfad2a"}
Aug 04 04:15:13.022220 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:13.022101    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb"
Aug 04 04:15:13.031858 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:13.031826    2579 scope.go:117] "RemoveContainer" containerID="068061a252e1161aebe351e0db010548384f71a95a4685d033978b6a0d8a5129"
Aug 04 04:15:13.048139 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:13.048109    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx"]
Aug 04 04:15:13.050089 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:13.050070    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d8d0c-8d9d8f796-8nmxx"]
Aug 04 04:15:13.060284 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:13.060264    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb"]
Aug 04 04:15:13.064330 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:13.064300    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d8d0c-7f97c589f6-bcdlb"]
Aug 04 04:15:13.845988 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:13.845929    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cdb62bc-26d6-42dd-ab6c-93c462d537b2" path="/var/lib/kubelet/pods/4cdb62bc-26d6-42dd-ab6c-93c462d537b2/volumes"
Aug 04 04:15:13.846465 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:13.846442    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e57d680-e609-4c20-ac5e-9bb7ae9bf560" path="/var/lib/kubelet/pods/4e57d680-e609-4c20-ac5e-9bb7ae9bf560/volumes"
Aug 04 04:15:13.846828 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:15:13.846811    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e76c67ae-9ab0-4e3f-990d-e5a11a736e21" path="/var/lib/kubelet/pods/e76c67ae-9ab0-4e3f-990d-e5a11a736e21/volumes"
Aug 04 04:16:11.689342 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:11.689308    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"]
Aug 04 04:16:11.690009 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:11.689541    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg" podUID="27bd5e08-1294-4ef1-a392-a6ca80f30d99" containerName="maas-api" containerID="cri-o://a8a189f0b870e200546f1b5725edfaa85dee8539cbc1cc4f919118301e86f4aa" gracePeriod=30
Aug 04 04:16:11.755449 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:11.755413    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g"]
Aug 04 04:16:11.755926 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:11.755889    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g" podUID="04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19" containerName="payload-processing" containerID="cri-o://d69e7048da7ffa97dafcdf8f3f815f8ab5accd0ccbfebcf9c9dd0c543fe482b4" gracePeriod=30
Aug 04 04:16:11.761089 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:11.761044    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn"]
Aug 04 04:16:11.761348 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:11.761298    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn" podUID="fdd912fe-891f-4b8b-994c-fd8baa7c6f87" containerName="payload-pre-processing" containerID="cri-o://650a035ace15b60e838b10f4950509027e161e3d83e58c0c8b0a663c8227f3ae" gracePeriod=30
Aug 04 04:16:12.024378 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.024345    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"
Aug 04 04:16:12.062996 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.062940    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g"
Aug 04 04:16:12.069111 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.069083    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt4jp\" (UniqueName: \"kubernetes.io/projected/27bd5e08-1294-4ef1-a392-a6ca80f30d99-kube-api-access-bt4jp\") pod \"27bd5e08-1294-4ef1-a392-a6ca80f30d99\" (UID: \"27bd5e08-1294-4ef1-a392-a6ca80f30d99\") "
Aug 04 04:16:12.069265 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.069119    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27bd5e08-1294-4ef1-a392-a6ca80f30d99-openshift-service-ca\") pod \"27bd5e08-1294-4ef1-a392-a6ca80f30d99\" (UID: \"27bd5e08-1294-4ef1-a392-a6ca80f30d99\") "
Aug 04 04:16:12.069265 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.069250    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27bd5e08-1294-4ef1-a392-a6ca80f30d99-maas-api-tls\") pod \"27bd5e08-1294-4ef1-a392-a6ca80f30d99\" (UID: \"27bd5e08-1294-4ef1-a392-a6ca80f30d99\") "
Aug 04 04:16:12.069556 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.069530    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27bd5e08-1294-4ef1-a392-a6ca80f30d99-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "27bd5e08-1294-4ef1-a392-a6ca80f30d99" (UID: "27bd5e08-1294-4ef1-a392-a6ca80f30d99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 04:16:12.071314 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.071290    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27bd5e08-1294-4ef1-a392-a6ca80f30d99-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "27bd5e08-1294-4ef1-a392-a6ca80f30d99" (UID: "27bd5e08-1294-4ef1-a392-a6ca80f30d99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 04:16:12.071418 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.071332    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27bd5e08-1294-4ef1-a392-a6ca80f30d99-kube-api-access-bt4jp" (OuterVolumeSpecName: "kube-api-access-bt4jp") pod "27bd5e08-1294-4ef1-a392-a6ca80f30d99" (UID: "27bd5e08-1294-4ef1-a392-a6ca80f30d99"). InnerVolumeSpecName "kube-api-access-bt4jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:16:12.075174 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.075153    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn"
Aug 04 04:16:12.170147 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.170109    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd912fe-891f-4b8b-994c-fd8baa7c6f87-plugins-config-volume\") pod \"fdd912fe-891f-4b8b-994c-fd8baa7c6f87\" (UID: \"fdd912fe-891f-4b8b-994c-fd8baa7c6f87\") "
Aug 04 04:16:12.170328 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.170236    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grwj5\" (UniqueName: \"kubernetes.io/projected/fdd912fe-891f-4b8b-994c-fd8baa7c6f87-kube-api-access-grwj5\") pod \"fdd912fe-891f-4b8b-994c-fd8baa7c6f87\" (UID: \"fdd912fe-891f-4b8b-994c-fd8baa7c6f87\") "
Aug 04 04:16:12.170328 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.170270    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8gj4\" (UniqueName: \"kubernetes.io/projected/04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19-kube-api-access-q8gj4\") pod \"04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19\" (UID: \"04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19\") "
Aug 04 04:16:12.170328 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.170288    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19-plugins-config-volume\") pod \"04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19\" (UID: \"04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19\") "
Aug 04 04:16:12.170557 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.170539    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27bd5e08-1294-4ef1-a392-a6ca80f30d99-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:16:12.170600 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.170563    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt4jp\" (UniqueName: \"kubernetes.io/projected/27bd5e08-1294-4ef1-a392-a6ca80f30d99-kube-api-access-bt4jp\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:16:12.170600 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.170580    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27bd5e08-1294-4ef1-a392-a6ca80f30d99-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:16:12.170600 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.170554    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd912fe-891f-4b8b-994c-fd8baa7c6f87-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fdd912fe-891f-4b8b-994c-fd8baa7c6f87" (UID: "fdd912fe-891f-4b8b-994c-fd8baa7c6f87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 04:16:12.170750 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.170729    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19" (UID: "04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 04:16:12.172365 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.172345    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd912fe-891f-4b8b-994c-fd8baa7c6f87-kube-api-access-grwj5" (OuterVolumeSpecName: "kube-api-access-grwj5") pod "fdd912fe-891f-4b8b-994c-fd8baa7c6f87" (UID: "fdd912fe-891f-4b8b-994c-fd8baa7c6f87"). InnerVolumeSpecName "kube-api-access-grwj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:16:12.172420 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.172370    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19-kube-api-access-q8gj4" (OuterVolumeSpecName: "kube-api-access-q8gj4") pod "04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19" (UID: "04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19"). InnerVolumeSpecName "kube-api-access-q8gj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:16:12.271179 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.271091    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grwj5\" (UniqueName: \"kubernetes.io/projected/fdd912fe-891f-4b8b-994c-fd8baa7c6f87-kube-api-access-grwj5\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:16:12.271179 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.271120    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8gj4\" (UniqueName: \"kubernetes.io/projected/04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19-kube-api-access-q8gj4\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:16:12.271179 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.271130    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:16:12.271179 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.271139    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd912fe-891f-4b8b-994c-fd8baa7c6f87-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:16:12.359970 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.359935    2579 generic.go:356] "Generic (PLEG): container finished" podID="27bd5e08-1294-4ef1-a392-a6ca80f30d99" containerID="a8a189f0b870e200546f1b5725edfaa85dee8539cbc1cc4f919118301e86f4aa" exitCode=0
Aug 04 04:16:12.360159 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.359995    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"
Aug 04 04:16:12.360159 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.360021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg" event={"ID":"27bd5e08-1294-4ef1-a392-a6ca80f30d99","Type":"ContainerDied","Data":"a8a189f0b870e200546f1b5725edfaa85dee8539cbc1cc4f919118301e86f4aa"}
Aug 04 04:16:12.360159 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.360079    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg" event={"ID":"27bd5e08-1294-4ef1-a392-a6ca80f30d99","Type":"ContainerDied","Data":"0da0e69a67c9f595e699b385b9ccee1a1e7ed83ad7df8feee339f57d0fb156c4"}
Aug 04 04:16:12.360159 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.360099    2579 scope.go:117] "RemoveContainer" containerID="a8a189f0b870e200546f1b5725edfaa85dee8539cbc1cc4f919118301e86f4aa"
Aug 04 04:16:12.361320 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.361296    2579 generic.go:356] "Generic (PLEG): container finished" podID="fdd912fe-891f-4b8b-994c-fd8baa7c6f87" containerID="650a035ace15b60e838b10f4950509027e161e3d83e58c0c8b0a663c8227f3ae" exitCode=0
Aug 04 04:16:12.361412 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.361340    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn"
Aug 04 04:16:12.361412 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.361379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn" event={"ID":"fdd912fe-891f-4b8b-994c-fd8baa7c6f87","Type":"ContainerDied","Data":"650a035ace15b60e838b10f4950509027e161e3d83e58c0c8b0a663c8227f3ae"}
Aug 04 04:16:12.361525 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.361421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn" event={"ID":"fdd912fe-891f-4b8b-994c-fd8baa7c6f87","Type":"ContainerDied","Data":"3620dddb202d70a55967a3f395d85e6571f87d549344bf76da6a788b02db624c"}
Aug 04 04:16:12.362790 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.362768    2579 generic.go:356] "Generic (PLEG): container finished" podID="04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19" containerID="d69e7048da7ffa97dafcdf8f3f815f8ab5accd0ccbfebcf9c9dd0c543fe482b4" exitCode=0
Aug 04 04:16:12.362909 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.362839    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g"
Aug 04 04:16:12.362909 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.362871    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g" event={"ID":"04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19","Type":"ContainerDied","Data":"d69e7048da7ffa97dafcdf8f3f815f8ab5accd0ccbfebcf9c9dd0c543fe482b4"}
Aug 04 04:16:12.363064 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.362920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g" event={"ID":"04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19","Type":"ContainerDied","Data":"098431aac776ec1e46972684aa76ba6f1fe6abf613ec5cad55a467bf4a370354"}
Aug 04 04:16:12.369877 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.369857    2579 scope.go:117] "RemoveContainer" containerID="a8a189f0b870e200546f1b5725edfaa85dee8539cbc1cc4f919118301e86f4aa"
Aug 04 04:16:12.370258 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:16:12.370229    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a189f0b870e200546f1b5725edfaa85dee8539cbc1cc4f919118301e86f4aa\": container with ID starting with a8a189f0b870e200546f1b5725edfaa85dee8539cbc1cc4f919118301e86f4aa not found: ID does not exist" containerID="a8a189f0b870e200546f1b5725edfaa85dee8539cbc1cc4f919118301e86f4aa"
Aug 04 04:16:12.370351 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.370269    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a189f0b870e200546f1b5725edfaa85dee8539cbc1cc4f919118301e86f4aa"} err="failed to get container status \"a8a189f0b870e200546f1b5725edfaa85dee8539cbc1cc4f919118301e86f4aa\": rpc error: code = NotFound desc = could not find container \"a8a189f0b870e200546f1b5725edfaa85dee8539cbc1cc4f919118301e86f4aa\": container with ID starting with a8a189f0b870e200546f1b5725edfaa85dee8539cbc1cc4f919118301e86f4aa not found: ID does not exist"
Aug 04 04:16:12.370351 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.370304    2579 scope.go:117] "RemoveContainer" containerID="650a035ace15b60e838b10f4950509027e161e3d83e58c0c8b0a663c8227f3ae"
Aug 04 04:16:12.379991 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.379945    2579 scope.go:117] "RemoveContainer" containerID="650a035ace15b60e838b10f4950509027e161e3d83e58c0c8b0a663c8227f3ae"
Aug 04 04:16:12.380287 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:16:12.380267    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650a035ace15b60e838b10f4950509027e161e3d83e58c0c8b0a663c8227f3ae\": container with ID starting with 650a035ace15b60e838b10f4950509027e161e3d83e58c0c8b0a663c8227f3ae not found: ID does not exist" containerID="650a035ace15b60e838b10f4950509027e161e3d83e58c0c8b0a663c8227f3ae"
Aug 04 04:16:12.380354 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.380296    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650a035ace15b60e838b10f4950509027e161e3d83e58c0c8b0a663c8227f3ae"} err="failed to get container status \"650a035ace15b60e838b10f4950509027e161e3d83e58c0c8b0a663c8227f3ae\": rpc error: code = NotFound desc = could not find container \"650a035ace15b60e838b10f4950509027e161e3d83e58c0c8b0a663c8227f3ae\": container with ID starting with 650a035ace15b60e838b10f4950509027e161e3d83e58c0c8b0a663c8227f3ae not found: ID does not exist"
Aug 04 04:16:12.380354 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.380315    2579 scope.go:117] "RemoveContainer" containerID="d69e7048da7ffa97dafcdf8f3f815f8ab5accd0ccbfebcf9c9dd0c543fe482b4"
Aug 04 04:16:12.388361 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.388335    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn"]
Aug 04 04:16:12.388935 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.388922    2579 scope.go:117] "RemoveContainer" containerID="d69e7048da7ffa97dafcdf8f3f815f8ab5accd0ccbfebcf9c9dd0c543fe482b4"
Aug 04 04:16:12.389253 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:16:12.389236    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d69e7048da7ffa97dafcdf8f3f815f8ab5accd0ccbfebcf9c9dd0c543fe482b4\": container with ID starting with d69e7048da7ffa97dafcdf8f3f815f8ab5accd0ccbfebcf9c9dd0c543fe482b4 not found: ID does not exist" containerID="d69e7048da7ffa97dafcdf8f3f815f8ab5accd0ccbfebcf9c9dd0c543fe482b4"
Aug 04 04:16:12.389305 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.389261    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d69e7048da7ffa97dafcdf8f3f815f8ab5accd0ccbfebcf9c9dd0c543fe482b4"} err="failed to get container status \"d69e7048da7ffa97dafcdf8f3f815f8ab5accd0ccbfebcf9c9dd0c543fe482b4\": rpc error: code = NotFound desc = could not find container \"d69e7048da7ffa97dafcdf8f3f815f8ab5accd0ccbfebcf9c9dd0c543fe482b4\": container with ID starting with d69e7048da7ffa97dafcdf8f3f815f8ab5accd0ccbfebcf9c9dd0c543fe482b4 not found: ID does not exist"
Aug 04 04:16:12.392199 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.392180    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0089f-7749b44d5b-mkwwn"]
Aug 04 04:16:12.403955 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.403932    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"]
Aug 04 04:16:12.409681 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.409660    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0089f-785c67c4f4-hpcvg"]
Aug 04 04:16:12.419556 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.419525    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g"]
Aug 04 04:16:12.424240 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:12.424219    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0089f-9b5fd8994-jtz8g"]
Aug 04 04:16:13.845386 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:13.845346    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19" path="/var/lib/kubelet/pods/04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19/volumes"
Aug 04 04:16:13.845826 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:13.845808    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27bd5e08-1294-4ef1-a392-a6ca80f30d99" path="/var/lib/kubelet/pods/27bd5e08-1294-4ef1-a392-a6ca80f30d99/volumes"
Aug 04 04:16:13.846217 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:13.846200    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdd912fe-891f-4b8b-994c-fd8baa7c6f87" path="/var/lib/kubelet/pods/fdd912fe-891f-4b8b-994c-fd8baa7c6f87/volumes"
Aug 04 04:16:29.525236 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:29.525200    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"]
Aug 04 04:16:29.525653 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:29.525406    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4" podUID="1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2" containerName="maas-api" containerID="cri-o://611ecaefd7ba2d25bb77fb0c955ecdfd26ed75b1ca224f4d4fbd82c4c8392da4" gracePeriod=30
Aug 04 04:16:29.584594 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:29.584557    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b"]
Aug 04 04:16:29.585113 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:29.584856    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b" podUID="683fc823-1d0c-46e6-addb-31a35d0cc710" containerName="payload-pre-processing" containerID="cri-o://ec1cfbaa5d128f1d674c190dee313807b5f74e734d264f987b167e60f5870c41" gracePeriod=30
Aug 04 04:16:29.834498 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:29.834475    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"
Aug 04 04:16:29.867456 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:29.867427    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b"
Aug 04 04:16:29.936049 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:29.936016    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-openshift-service-ca\") pod \"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2\" (UID: \"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2\") "
Aug 04 04:16:29.936212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:29.936056    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m55pg\" (UniqueName: \"kubernetes.io/projected/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-kube-api-access-m55pg\") pod \"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2\" (UID: \"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2\") "
Aug 04 04:16:29.936212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:29.936123    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-maas-api-tls\") pod \"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2\" (UID: \"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2\") "
Aug 04 04:16:29.936449 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:29.936427    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2" (UID: "1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 04:16:29.938161 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:29.938146    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2" (UID: "1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 04:16:29.938228 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:29.938206    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-kube-api-access-m55pg" (OuterVolumeSpecName: "kube-api-access-m55pg") pod "1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2" (UID: "1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2"). InnerVolumeSpecName "kube-api-access-m55pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:16:30.036829 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.036795    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gv5d9\" (UniqueName: \"kubernetes.io/projected/683fc823-1d0c-46e6-addb-31a35d0cc710-kube-api-access-gv5d9\") pod \"683fc823-1d0c-46e6-addb-31a35d0cc710\" (UID: \"683fc823-1d0c-46e6-addb-31a35d0cc710\") "
Aug 04 04:16:30.037006 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.036912    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/683fc823-1d0c-46e6-addb-31a35d0cc710-plugins-config-volume\") pod \"683fc823-1d0c-46e6-addb-31a35d0cc710\" (UID: \"683fc823-1d0c-46e6-addb-31a35d0cc710\") "
Aug 04 04:16:30.037154 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.037142    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:16:30.037195 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.037163    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:16:30.037195 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.037179    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m55pg\" (UniqueName: \"kubernetes.io/projected/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2-kube-api-access-m55pg\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:16:30.037316 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.037295    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/683fc823-1d0c-46e6-addb-31a35d0cc710-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "683fc823-1d0c-46e6-addb-31a35d0cc710" (UID: "683fc823-1d0c-46e6-addb-31a35d0cc710"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 04:16:30.038989 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.038969    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683fc823-1d0c-46e6-addb-31a35d0cc710-kube-api-access-gv5d9" (OuterVolumeSpecName: "kube-api-access-gv5d9") pod "683fc823-1d0c-46e6-addb-31a35d0cc710" (UID: "683fc823-1d0c-46e6-addb-31a35d0cc710"). InnerVolumeSpecName "kube-api-access-gv5d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:16:30.138026 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.137937    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/683fc823-1d0c-46e6-addb-31a35d0cc710-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:16:30.138026 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.137991    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv5d9\" (UniqueName: \"kubernetes.io/projected/683fc823-1d0c-46e6-addb-31a35d0cc710-kube-api-access-gv5d9\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:16:30.472387 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.472352    2579 generic.go:356] "Generic (PLEG): container finished" podID="683fc823-1d0c-46e6-addb-31a35d0cc710" containerID="ec1cfbaa5d128f1d674c190dee313807b5f74e734d264f987b167e60f5870c41" exitCode=0
Aug 04 04:16:30.472590 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.472401    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b"
Aug 04 04:16:30.472590 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.472428    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b" event={"ID":"683fc823-1d0c-46e6-addb-31a35d0cc710","Type":"ContainerDied","Data":"ec1cfbaa5d128f1d674c190dee313807b5f74e734d264f987b167e60f5870c41"}
Aug 04 04:16:30.472590 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.472477    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b" event={"ID":"683fc823-1d0c-46e6-addb-31a35d0cc710","Type":"ContainerDied","Data":"246981a600d896c4d98a04cd6b25dee16aad747de948e48e76861cea2e85ae7b"}
Aug 04 04:16:30.472590 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.472509    2579 scope.go:117] "RemoveContainer" containerID="ec1cfbaa5d128f1d674c190dee313807b5f74e734d264f987b167e60f5870c41"
Aug 04 04:16:30.473805 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.473782    2579 generic.go:356] "Generic (PLEG): container finished" podID="1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2" containerID="611ecaefd7ba2d25bb77fb0c955ecdfd26ed75b1ca224f4d4fbd82c4c8392da4" exitCode=0
Aug 04 04:16:30.473894 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.473833    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"
Aug 04 04:16:30.473894 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.473866    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4" event={"ID":"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2","Type":"ContainerDied","Data":"611ecaefd7ba2d25bb77fb0c955ecdfd26ed75b1ca224f4d4fbd82c4c8392da4"}
Aug 04 04:16:30.474017 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.473926    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4" event={"ID":"1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2","Type":"ContainerDied","Data":"417eff3d8ca5ecc438439a0a65e493117789cab314d4e798d20b2a5dc10f13a0"}
Aug 04 04:16:30.481560 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.481544    2579 scope.go:117] "RemoveContainer" containerID="ec1cfbaa5d128f1d674c190dee313807b5f74e734d264f987b167e60f5870c41"
Aug 04 04:16:30.482127 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:16:30.481885    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec1cfbaa5d128f1d674c190dee313807b5f74e734d264f987b167e60f5870c41\": container with ID starting with ec1cfbaa5d128f1d674c190dee313807b5f74e734d264f987b167e60f5870c41 not found: ID does not exist" containerID="ec1cfbaa5d128f1d674c190dee313807b5f74e734d264f987b167e60f5870c41"
Aug 04 04:16:30.482127 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.481911    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1cfbaa5d128f1d674c190dee313807b5f74e734d264f987b167e60f5870c41"} err="failed to get container status \"ec1cfbaa5d128f1d674c190dee313807b5f74e734d264f987b167e60f5870c41\": rpc error: code = NotFound desc = could not find container \"ec1cfbaa5d128f1d674c190dee313807b5f74e734d264f987b167e60f5870c41\": container with ID starting with ec1cfbaa5d128f1d674c190dee313807b5f74e734d264f987b167e60f5870c41 not found: ID does not exist"
Aug 04 04:16:30.482127 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.481944    2579 scope.go:117] "RemoveContainer" containerID="611ecaefd7ba2d25bb77fb0c955ecdfd26ed75b1ca224f4d4fbd82c4c8392da4"
Aug 04 04:16:30.490667 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.490648    2579 scope.go:117] "RemoveContainer" containerID="611ecaefd7ba2d25bb77fb0c955ecdfd26ed75b1ca224f4d4fbd82c4c8392da4"
Aug 04 04:16:30.490952 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:16:30.490932    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"611ecaefd7ba2d25bb77fb0c955ecdfd26ed75b1ca224f4d4fbd82c4c8392da4\": container with ID starting with 611ecaefd7ba2d25bb77fb0c955ecdfd26ed75b1ca224f4d4fbd82c4c8392da4 not found: ID does not exist" containerID="611ecaefd7ba2d25bb77fb0c955ecdfd26ed75b1ca224f4d4fbd82c4c8392da4"
Aug 04 04:16:30.491032 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.490979    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611ecaefd7ba2d25bb77fb0c955ecdfd26ed75b1ca224f4d4fbd82c4c8392da4"} err="failed to get container status \"611ecaefd7ba2d25bb77fb0c955ecdfd26ed75b1ca224f4d4fbd82c4c8392da4\": rpc error: code = NotFound desc = could not find container \"611ecaefd7ba2d25bb77fb0c955ecdfd26ed75b1ca224f4d4fbd82c4c8392da4\": container with ID starting with 611ecaefd7ba2d25bb77fb0c955ecdfd26ed75b1ca224f4d4fbd82c4c8392da4 not found: ID does not exist"
Aug 04 04:16:30.499816 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.499790    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"]
Aug 04 04:16:30.506689 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.506666    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-07c848-7f8bb578cb-vv6r4"]
Aug 04 04:16:30.516665 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.516635    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b"]
Aug 04 04:16:30.520841 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:30.520821    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-07c848-5846f7c584-q8l5b"]
Aug 04 04:16:31.845223 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:31.845192    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2" path="/var/lib/kubelet/pods/1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2/volumes"
Aug 04 04:16:31.845747 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:31.845725    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="683fc823-1d0c-46e6-addb-31a35d0cc710" path="/var/lib/kubelet/pods/683fc823-1d0c-46e6-addb-31a35d0cc710/volumes"
Aug 04 04:16:47.145917 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.145880    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"]
Aug 04 04:16:47.146474 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.146104    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz" podUID="667d9804-3048-46b4-9eda-03bb5a3febbc" containerName="maas-api" containerID="cri-o://c8cbc4779bc9b90dd48802db4fc41273ef61a08f0b087fd3db2da3ee2756ccb5" gracePeriod=30
Aug 04 04:16:47.263391 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.263352    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn"]
Aug 04 04:16:47.263848 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.263638    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn" podUID="fe53c783-5a46-4df1-8678-3bac9b39849b" containerName="payload-processing" containerID="cri-o://c0a3f27e6d11a4fd53005c275d626d18f8d5591143f25262a219883a93d23642" gracePeriod=30
Aug 04 04:16:47.271550 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.271496    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2"]
Aug 04 04:16:47.271752 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.271722    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2" podUID="6ae535cf-c48a-491b-94e3-4188808e06e4" containerName="payload-pre-processing" containerID="cri-o://7bc1254506ddc9a76402a18a108c6524707dab0b2f1a6513d1c1689025c68073" gracePeriod=30
Aug 04 04:16:47.429006 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.428950    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"
Aug 04 04:16:47.501505 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.501471    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/667d9804-3048-46b4-9eda-03bb5a3febbc-openshift-service-ca\") pod \"667d9804-3048-46b4-9eda-03bb5a3febbc\" (UID: \"667d9804-3048-46b4-9eda-03bb5a3febbc\") "
Aug 04 04:16:47.501679 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.501596    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhf46\" (UniqueName: \"kubernetes.io/projected/667d9804-3048-46b4-9eda-03bb5a3febbc-kube-api-access-vhf46\") pod \"667d9804-3048-46b4-9eda-03bb5a3febbc\" (UID: \"667d9804-3048-46b4-9eda-03bb5a3febbc\") "
Aug 04 04:16:47.501679 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.501644    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/667d9804-3048-46b4-9eda-03bb5a3febbc-maas-api-tls\") pod \"667d9804-3048-46b4-9eda-03bb5a3febbc\" (UID: \"667d9804-3048-46b4-9eda-03bb5a3febbc\") "
Aug 04 04:16:47.502380 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.502326    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667d9804-3048-46b4-9eda-03bb5a3febbc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "667d9804-3048-46b4-9eda-03bb5a3febbc" (UID: "667d9804-3048-46b4-9eda-03bb5a3febbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 04:16:47.505481 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.505451    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/667d9804-3048-46b4-9eda-03bb5a3febbc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "667d9804-3048-46b4-9eda-03bb5a3febbc" (UID: "667d9804-3048-46b4-9eda-03bb5a3febbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 04:16:47.505648 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.505621    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667d9804-3048-46b4-9eda-03bb5a3febbc-kube-api-access-vhf46" (OuterVolumeSpecName: "kube-api-access-vhf46") pod "667d9804-3048-46b4-9eda-03bb5a3febbc" (UID: "667d9804-3048-46b4-9eda-03bb5a3febbc"). InnerVolumeSpecName "kube-api-access-vhf46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:16:47.540579 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.540551    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn"
Aug 04 04:16:47.543771 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.543750    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2"
Aug 04 04:16:47.561090 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.561054    2579 generic.go:356] "Generic (PLEG): container finished" podID="6ae535cf-c48a-491b-94e3-4188808e06e4" containerID="7bc1254506ddc9a76402a18a108c6524707dab0b2f1a6513d1c1689025c68073" exitCode=0
Aug 04 04:16:47.561239 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.561139    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2"
Aug 04 04:16:47.561366 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.561134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2" event={"ID":"6ae535cf-c48a-491b-94e3-4188808e06e4","Type":"ContainerDied","Data":"7bc1254506ddc9a76402a18a108c6524707dab0b2f1a6513d1c1689025c68073"}
Aug 04 04:16:47.561554 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.561530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2" event={"ID":"6ae535cf-c48a-491b-94e3-4188808e06e4","Type":"ContainerDied","Data":"d3f5ec73f7063da5d0ee5646405a732c7a027eaa621d648688f8dead3343559c"}
Aug 04 04:16:47.561669 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.561574    2579 scope.go:117] "RemoveContainer" containerID="7bc1254506ddc9a76402a18a108c6524707dab0b2f1a6513d1c1689025c68073"
Aug 04 04:16:47.564006 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.563000    2579 generic.go:356] "Generic (PLEG): container finished" podID="667d9804-3048-46b4-9eda-03bb5a3febbc" containerID="c8cbc4779bc9b90dd48802db4fc41273ef61a08f0b087fd3db2da3ee2756ccb5" exitCode=0
Aug 04 04:16:47.564006 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.563078    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz" event={"ID":"667d9804-3048-46b4-9eda-03bb5a3febbc","Type":"ContainerDied","Data":"c8cbc4779bc9b90dd48802db4fc41273ef61a08f0b087fd3db2da3ee2756ccb5"}
Aug 04 04:16:47.564006 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.563128    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz" event={"ID":"667d9804-3048-46b4-9eda-03bb5a3febbc","Type":"ContainerDied","Data":"e2cebc369cfe6e0f43ea7f6642eb979c39aa0a3d72d1f923d4bd0e13a55c7ae1"}
Aug 04 04:16:47.564006 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.563189    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"
Aug 04 04:16:47.564933 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.564903    2579 generic.go:356] "Generic (PLEG): container finished" podID="fe53c783-5a46-4df1-8678-3bac9b39849b" containerID="c0a3f27e6d11a4fd53005c275d626d18f8d5591143f25262a219883a93d23642" exitCode=0
Aug 04 04:16:47.565053 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.564955    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn"
Aug 04 04:16:47.565115 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.565046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn" event={"ID":"fe53c783-5a46-4df1-8678-3bac9b39849b","Type":"ContainerDied","Data":"c0a3f27e6d11a4fd53005c275d626d18f8d5591143f25262a219883a93d23642"}
Aug 04 04:16:47.565115 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.565100    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn" event={"ID":"fe53c783-5a46-4df1-8678-3bac9b39849b","Type":"ContainerDied","Data":"65d2b673cbc4785f36f8df4eae09d5604b84b554157253eb5ead80562f036241"}
Aug 04 04:16:47.575178 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.574769    2579 scope.go:117] "RemoveContainer" containerID="7bc1254506ddc9a76402a18a108c6524707dab0b2f1a6513d1c1689025c68073"
Aug 04 04:16:47.575322 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:16:47.575250    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc1254506ddc9a76402a18a108c6524707dab0b2f1a6513d1c1689025c68073\": container with ID starting with 7bc1254506ddc9a76402a18a108c6524707dab0b2f1a6513d1c1689025c68073 not found: ID does not exist" containerID="7bc1254506ddc9a76402a18a108c6524707dab0b2f1a6513d1c1689025c68073"
Aug 04 04:16:47.575322 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.575281    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc1254506ddc9a76402a18a108c6524707dab0b2f1a6513d1c1689025c68073"} err="failed to get container status \"7bc1254506ddc9a76402a18a108c6524707dab0b2f1a6513d1c1689025c68073\": rpc error: code = NotFound desc = could not find container \"7bc1254506ddc9a76402a18a108c6524707dab0b2f1a6513d1c1689025c68073\": container with ID starting with 7bc1254506ddc9a76402a18a108c6524707dab0b2f1a6513d1c1689025c68073 not found: ID does not exist"
Aug 04 04:16:47.575322 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.575312    2579 scope.go:117] "RemoveContainer" containerID="c8cbc4779bc9b90dd48802db4fc41273ef61a08f0b087fd3db2da3ee2756ccb5"
Aug 04 04:16:47.585801 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.585784    2579 scope.go:117] "RemoveContainer" containerID="c8cbc4779bc9b90dd48802db4fc41273ef61a08f0b087fd3db2da3ee2756ccb5"
Aug 04 04:16:47.586119 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:16:47.586098    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8cbc4779bc9b90dd48802db4fc41273ef61a08f0b087fd3db2da3ee2756ccb5\": container with ID starting with c8cbc4779bc9b90dd48802db4fc41273ef61a08f0b087fd3db2da3ee2756ccb5 not found: ID does not exist" containerID="c8cbc4779bc9b90dd48802db4fc41273ef61a08f0b087fd3db2da3ee2756ccb5"
Aug 04 04:16:47.586212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.586124    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8cbc4779bc9b90dd48802db4fc41273ef61a08f0b087fd3db2da3ee2756ccb5"} err="failed to get container status \"c8cbc4779bc9b90dd48802db4fc41273ef61a08f0b087fd3db2da3ee2756ccb5\": rpc error: code = NotFound desc = could not find container \"c8cbc4779bc9b90dd48802db4fc41273ef61a08f0b087fd3db2da3ee2756ccb5\": container with ID starting with c8cbc4779bc9b90dd48802db4fc41273ef61a08f0b087fd3db2da3ee2756ccb5 not found: ID does not exist"
Aug 04 04:16:47.586212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.586147    2579 scope.go:117] "RemoveContainer" containerID="c0a3f27e6d11a4fd53005c275d626d18f8d5591143f25262a219883a93d23642"
Aug 04 04:16:47.595601 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.595576    2579 scope.go:117] "RemoveContainer" containerID="c0a3f27e6d11a4fd53005c275d626d18f8d5591143f25262a219883a93d23642"
Aug 04 04:16:47.595911 ip-10-0-138-99 kubenswrapper[2579]: E0804 04:16:47.595884    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a3f27e6d11a4fd53005c275d626d18f8d5591143f25262a219883a93d23642\": container with ID starting with c0a3f27e6d11a4fd53005c275d626d18f8d5591143f25262a219883a93d23642 not found: ID does not exist" containerID="c0a3f27e6d11a4fd53005c275d626d18f8d5591143f25262a219883a93d23642"
Aug 04 04:16:47.596103 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.595924    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a3f27e6d11a4fd53005c275d626d18f8d5591143f25262a219883a93d23642"} err="failed to get container status \"c0a3f27e6d11a4fd53005c275d626d18f8d5591143f25262a219883a93d23642\": rpc error: code = NotFound desc = could not find container \"c0a3f27e6d11a4fd53005c275d626d18f8d5591143f25262a219883a93d23642\": container with ID starting with c0a3f27e6d11a4fd53005c275d626d18f8d5591143f25262a219883a93d23642 not found: ID does not exist"
Aug 04 04:16:47.598503 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.598478    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"]
Aug 04 04:16:47.600925 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.600904    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-881c6f-c6cc57b45-sv4wz"]
Aug 04 04:16:47.602236 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.602221    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae535cf-c48a-491b-94e3-4188808e06e4-plugins-config-volume\") pod \"6ae535cf-c48a-491b-94e3-4188808e06e4\" (UID: \"6ae535cf-c48a-491b-94e3-4188808e06e4\") "
Aug 04 04:16:47.602316 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.602256    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp75n\" (UniqueName: \"kubernetes.io/projected/fe53c783-5a46-4df1-8678-3bac9b39849b-kube-api-access-wp75n\") pod \"fe53c783-5a46-4df1-8678-3bac9b39849b\" (UID: \"fe53c783-5a46-4df1-8678-3bac9b39849b\") "
Aug 04 04:16:47.602316 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.602281    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcfwx\" (UniqueName: \"kubernetes.io/projected/6ae535cf-c48a-491b-94e3-4188808e06e4-kube-api-access-vcfwx\") pod \"6ae535cf-c48a-491b-94e3-4188808e06e4\" (UID: \"6ae535cf-c48a-491b-94e3-4188808e06e4\") "
Aug 04 04:16:47.602420 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.602344    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe53c783-5a46-4df1-8678-3bac9b39849b-plugins-config-volume\") pod \"fe53c783-5a46-4df1-8678-3bac9b39849b\" (UID: \"fe53c783-5a46-4df1-8678-3bac9b39849b\") "
Aug 04 04:16:47.602642 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.602579    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/667d9804-3048-46b4-9eda-03bb5a3febbc-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:16:47.602642 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.602604    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/667d9804-3048-46b4-9eda-03bb5a3febbc-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:16:47.602642 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.602622    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhf46\" (UniqueName: \"kubernetes.io/projected/667d9804-3048-46b4-9eda-03bb5a3febbc-kube-api-access-vhf46\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:16:47.602853 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.602734    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe53c783-5a46-4df1-8678-3bac9b39849b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe53c783-5a46-4df1-8678-3bac9b39849b" (UID: "fe53c783-5a46-4df1-8678-3bac9b39849b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 04:16:47.602853 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.602762    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae535cf-c48a-491b-94e3-4188808e06e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ae535cf-c48a-491b-94e3-4188808e06e4" (UID: "6ae535cf-c48a-491b-94e3-4188808e06e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 04:16:47.604378 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.604358    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae535cf-c48a-491b-94e3-4188808e06e4-kube-api-access-vcfwx" (OuterVolumeSpecName: "kube-api-access-vcfwx") pod "6ae535cf-c48a-491b-94e3-4188808e06e4" (UID: "6ae535cf-c48a-491b-94e3-4188808e06e4"). InnerVolumeSpecName "kube-api-access-vcfwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:16:47.604459 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.604404    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe53c783-5a46-4df1-8678-3bac9b39849b-kube-api-access-wp75n" (OuterVolumeSpecName: "kube-api-access-wp75n") pod "fe53c783-5a46-4df1-8678-3bac9b39849b" (UID: "fe53c783-5a46-4df1-8678-3bac9b39849b"). InnerVolumeSpecName "kube-api-access-wp75n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 04:16:47.703509 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.703425    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae535cf-c48a-491b-94e3-4188808e06e4-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:16:47.703509 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.703453    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp75n\" (UniqueName: \"kubernetes.io/projected/fe53c783-5a46-4df1-8678-3bac9b39849b-kube-api-access-wp75n\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:16:47.703509 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.703464    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcfwx\" (UniqueName: \"kubernetes.io/projected/6ae535cf-c48a-491b-94e3-4188808e06e4-kube-api-access-vcfwx\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:16:47.703509 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.703472    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe53c783-5a46-4df1-8678-3bac9b39849b-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Aug 04 04:16:47.845707 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.845676    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="667d9804-3048-46b4-9eda-03bb5a3febbc" path="/var/lib/kubelet/pods/667d9804-3048-46b4-9eda-03bb5a3febbc/volumes"
Aug 04 04:16:47.876357 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.876326    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2"]
Aug 04 04:16:47.880504 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.880480    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-881c6f-d8869dcbd-ks9g2"]
Aug 04 04:16:47.894082 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.894054    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn"]
Aug 04 04:16:47.896501 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:47.896478    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-881c6f-79f965cffd-hx5gn"]
Aug 04 04:16:49.846133 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:49.846099    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ae535cf-c48a-491b-94e3-4188808e06e4" path="/var/lib/kubelet/pods/6ae535cf-c48a-491b-94e3-4188808e06e4/volumes"
Aug 04 04:16:49.846528 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:16:49.846485    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe53c783-5a46-4df1-8678-3bac9b39849b" path="/var/lib/kubelet/pods/fe53c783-5a46-4df1-8678-3bac9b39849b/volumes"
Aug 04 04:17:00.324795 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:00.324743    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5465dcb4d8-z7qrk_9a292744-1316-4511-bd59-4a1b67a2d714/manager/0.log"
Aug 04 04:17:00.950105 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:00.950073    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx_3c239d70-6373-43eb-acd1-2fb0a5631d82/copy-manifests/0.log"
Aug 04 04:17:00.956466 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:00.956435    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx_3c239d70-6373-43eb-acd1-2fb0a5631d82/manager/0.log"
Aug 04 04:17:01.302441 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:01.302383    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7cdb695bfd-7xmds_2e350ced-1eed-48f7-b2c3-bce2223db31a/maas-api/0.log"
Aug 04 04:17:01.751089 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:01.751048    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-twdj2_0caa0967-9ea2-4918-a068-d06269400223/postgres/0.log"
Aug 04 04:17:03.383188 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:03.383155    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-l6sx5_d3f95fe4-15d1-4821-a3dd-4f01792ccd30/kuadrant-console-plugin/0.log"
Aug 04 04:17:04.538199 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:04.538175    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69dbcc4dfb-lmwkc_4732185c-0a57-4163-b84b-89c007fbed25/kube-auth-proxy/0.log"
Aug 04 04:17:04.879457 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:04.879378    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-46wnj_2dbe90ed-02ed-428f-ad98-ae60193a9e66/payload-processing/0.log"
Aug 04 04:17:05.461570 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:05.461537    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-gvhmn_74775343-a44c-4b57-8430-d25999be311d/main/0.log"
Aug 04 04:17:05.571550 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:05.571523    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-vchxc_cd4a7161-856a-4ca1-8fbb-7f5ebb94b266/main/0.log"
Aug 04 04:17:14.486985 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:14.486927    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gcg5l_54b8b42f-6de1-4f5a-8115-50f8b2ff573f/global-pull-secret-syncer/0.log"
Aug 04 04:17:14.667560 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:14.667530    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v6hhn_cb1dbb4a-c31f-4485-b847-9a54379f39d1/konnectivity-agent/0.log"
Aug 04 04:17:14.751396 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:14.751321    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-99.ec2.internal_3fb028bcdaa2f5ddcb427d9e021fdc0b/haproxy/0.log"
Aug 04 04:17:20.388836 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:20.388803    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-l6sx5_d3f95fe4-15d1-4821-a3dd-4f01792ccd30/kuadrant-console-plugin/0.log"
Aug 04 04:17:22.354729 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:22.354703    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45c01697-5168-4b0e-ac9b-ca821db8c611/alertmanager/0.log"
Aug 04 04:17:22.377455 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:22.377425    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45c01697-5168-4b0e-ac9b-ca821db8c611/config-reloader/0.log"
Aug 04 04:17:22.399846 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:22.399819    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45c01697-5168-4b0e-ac9b-ca821db8c611/kube-rbac-proxy-web/0.log"
Aug 04 04:17:22.422374 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:22.422346    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45c01697-5168-4b0e-ac9b-ca821db8c611/kube-rbac-proxy/0.log"
Aug 04 04:17:22.446777 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:22.446750    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45c01697-5168-4b0e-ac9b-ca821db8c611/kube-rbac-proxy-metric/0.log"
Aug 04 04:17:22.472208 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:22.472186    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45c01697-5168-4b0e-ac9b-ca821db8c611/prom-label-proxy/0.log"
Aug 04 04:17:22.493113 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:22.493089    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45c01697-5168-4b0e-ac9b-ca821db8c611/init-config-reloader/0.log"
Aug 04 04:17:22.535128 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:22.535101    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-qgpdb_f2d6c41f-c453-40d7-be08-ea72a2fcf37d/cluster-monitoring-operator/0.log"
Aug 04 04:17:22.562177 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:22.562150    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-frz48_437947e9-5b4e-4466-ba95-79d38416d1fd/kube-state-metrics/0.log"
Aug 04 04:17:22.584946 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:22.584873    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-frz48_437947e9-5b4e-4466-ba95-79d38416d1fd/kube-rbac-proxy-main/0.log"
Aug 04 04:17:22.608238 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:22.608217    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-frz48_437947e9-5b4e-4466-ba95-79d38416d1fd/kube-rbac-proxy-self/0.log"
Aug 04 04:17:22.780396 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:22.780361    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xmjtq_8f46088b-d589-4008-ad8c-56c451ebb5b4/node-exporter/0.log"
Aug 04 04:17:22.803401 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:22.803374    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xmjtq_8f46088b-d589-4008-ad8c-56c451ebb5b4/kube-rbac-proxy/0.log"
Aug 04 04:17:22.830222 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:22.830192    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xmjtq_8f46088b-d589-4008-ad8c-56c451ebb5b4/init-textfile/0.log"
Aug 04 04:17:23.049139 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:23.049115    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0978753a-7c25-4ff0-8a6c-236e3727b4a8/prometheus/0.log"
Aug 04 04:17:23.073008 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:23.072978    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0978753a-7c25-4ff0-8a6c-236e3727b4a8/config-reloader/0.log"
Aug 04 04:17:23.098849 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:23.098825    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0978753a-7c25-4ff0-8a6c-236e3727b4a8/thanos-sidecar/0.log"
Aug 04 04:17:23.125513 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:23.125485    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0978753a-7c25-4ff0-8a6c-236e3727b4a8/kube-rbac-proxy-web/0.log"
Aug 04 04:17:23.148691 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:23.148624    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0978753a-7c25-4ff0-8a6c-236e3727b4a8/kube-rbac-proxy/0.log"
Aug 04 04:17:23.170929 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:23.170905    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0978753a-7c25-4ff0-8a6c-236e3727b4a8/kube-rbac-proxy-thanos/0.log"
Aug 04 04:17:23.199434 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:23.199412    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0978753a-7c25-4ff0-8a6c-236e3727b4a8/init-config-reloader/0.log"
Aug 04 04:17:23.273353 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:23.273325    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-9x8zj_57105572-98a4-43d9-98c7-ba99c117ad2e/prometheus-operator-admission-webhook/0.log"
Aug 04 04:17:24.757464 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:24.757437    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-2t9qv_9c78063e-e4f9-446b-90c5-89f0f2ddc001/networking-console-plugin/0.log"
Aug 04 04:17:25.473707 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.473675    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"]
Aug 04 04:17:25.474079 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474065    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="683fc823-1d0c-46e6-addb-31a35d0cc710" containerName="payload-pre-processing"
Aug 04 04:17:25.474079 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474080    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="683fc823-1d0c-46e6-addb-31a35d0cc710" containerName="payload-pre-processing"
Aug 04 04:17:25.474212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474090    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02852acd-9031-4aee-adb3-85396e768bf9" containerName="payload-processing"
Aug 04 04:17:25.474212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474096    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="02852acd-9031-4aee-adb3-85396e768bf9" containerName="payload-processing"
Aug 04 04:17:25.474212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474112    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ae535cf-c48a-491b-94e3-4188808e06e4" containerName="payload-pre-processing"
Aug 04 04:17:25.474212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474120    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ae535cf-c48a-491b-94e3-4188808e06e4" containerName="payload-pre-processing"
Aug 04 04:17:25.474212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474128    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd912fe-891f-4b8b-994c-fd8baa7c6f87" containerName="payload-pre-processing"
Aug 04 04:17:25.474212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474137    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd912fe-891f-4b8b-994c-fd8baa7c6f87" containerName="payload-pre-processing"
Aug 04 04:17:25.474212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474148    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19" containerName="payload-processing"
Aug 04 04:17:25.474212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474156    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19" containerName="payload-processing"
Aug 04 04:17:25.474212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474165    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2" containerName="maas-api"
Aug 04 04:17:25.474212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474170    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2" containerName="maas-api"
Aug 04 04:17:25.474212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474178    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cdb62bc-26d6-42dd-ab6c-93c462d537b2" containerName="payload-processing"
Aug 04 04:17:25.474212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474183    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cdb62bc-26d6-42dd-ab6c-93c462d537b2" containerName="payload-processing"
Aug 04 04:17:25.474212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474188    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="243466d7-23f2-4e47-9660-ab21ab9b8eed" containerName="maas-api"
Aug 04 04:17:25.474212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474193    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="243466d7-23f2-4e47-9660-ab21ab9b8eed" containerName="maas-api"
Aug 04 04:17:25.474212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474204    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e76c67ae-9ab0-4e3f-990d-e5a11a736e21" containerName="payload-pre-processing"
Aug 04 04:17:25.474212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474208    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e76c67ae-9ab0-4e3f-990d-e5a11a736e21" containerName="payload-pre-processing"
Aug 04 04:17:25.474212 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474215    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667d9804-3048-46b4-9eda-03bb5a3febbc" containerName="maas-api"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474223    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="667d9804-3048-46b4-9eda-03bb5a3febbc" containerName="maas-api"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474232    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe53c783-5a46-4df1-8678-3bac9b39849b" containerName="payload-processing"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474241    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe53c783-5a46-4df1-8678-3bac9b39849b" containerName="payload-processing"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474251    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27bd5e08-1294-4ef1-a392-a6ca80f30d99" containerName="maas-api"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474256    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="27bd5e08-1294-4ef1-a392-a6ca80f30d99" containerName="maas-api"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474264    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e57d680-e609-4c20-ac5e-9bb7ae9bf560" containerName="maas-api"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474269    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e57d680-e609-4c20-ac5e-9bb7ae9bf560" containerName="maas-api"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474341    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="02852acd-9031-4aee-adb3-85396e768bf9" containerName="payload-processing"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474350    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe53c783-5a46-4df1-8678-3bac9b39849b" containerName="payload-processing"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474357    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd912fe-891f-4b8b-994c-fd8baa7c6f87" containerName="payload-pre-processing"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474364    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ae535cf-c48a-491b-94e3-4188808e06e4" containerName="payload-pre-processing"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474372    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cdb62bc-26d6-42dd-ab6c-93c462d537b2" containerName="payload-processing"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474382    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fd6e00c-f5d1-4aa8-b955-2c295b5f33b2" containerName="maas-api"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474387    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e76c67ae-9ab0-4e3f-990d-e5a11a736e21" containerName="payload-pre-processing"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474394    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="667d9804-3048-46b4-9eda-03bb5a3febbc" containerName="maas-api"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474399    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e57d680-e609-4c20-ac5e-9bb7ae9bf560" containerName="maas-api"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474404    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="04cdf67b-8c92-4ddd-bbfc-3ceeffd75b19" containerName="payload-processing"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474410    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="27bd5e08-1294-4ef1-a392-a6ca80f30d99" containerName="maas-api"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474415    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="243466d7-23f2-4e47-9660-ab21ab9b8eed" containerName="maas-api"
Aug 04 04:17:25.474789 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.474421    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="683fc823-1d0c-46e6-addb-31a35d0cc710" containerName="payload-pre-processing"
Aug 04 04:17:25.477655 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.477639    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"
Aug 04 04:17:25.480241 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.480217    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nsl8p\"/\"kube-root-ca.crt\""
Aug 04 04:17:25.481104 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.481081    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nsl8p\"/\"default-dockercfg-c855t\""
Aug 04 04:17:25.481104 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.481091    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nsl8p\"/\"openshift-service-ca.crt\""
Aug 04 04:17:25.489917 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.489893    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"]
Aug 04 04:17:25.553208 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.553171    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj56c\" (UniqueName: \"kubernetes.io/projected/b4510463-becb-456a-8a84-5488f061a777-kube-api-access-zj56c\") pod \"perf-node-gather-daemonset-kxt5d\" (UID: \"b4510463-becb-456a-8a84-5488f061a777\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"
Aug 04 04:17:25.553208 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.553212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4510463-becb-456a-8a84-5488f061a777-sys\") pod \"perf-node-gather-daemonset-kxt5d\" (UID: \"b4510463-becb-456a-8a84-5488f061a777\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"
Aug 04 04:17:25.553477 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.553313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b4510463-becb-456a-8a84-5488f061a777-proc\") pod \"perf-node-gather-daemonset-kxt5d\" (UID: \"b4510463-becb-456a-8a84-5488f061a777\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"
Aug 04 04:17:25.553477 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.553371    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4510463-becb-456a-8a84-5488f061a777-lib-modules\") pod \"perf-node-gather-daemonset-kxt5d\" (UID: \"b4510463-becb-456a-8a84-5488f061a777\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"
Aug 04 04:17:25.553477 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.553431    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b4510463-becb-456a-8a84-5488f061a777-podres\") pod \"perf-node-gather-daemonset-kxt5d\" (UID: \"b4510463-becb-456a-8a84-5488f061a777\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"
Aug 04 04:17:25.654928 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.654885    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b4510463-becb-456a-8a84-5488f061a777-proc\") pod \"perf-node-gather-daemonset-kxt5d\" (UID: \"b4510463-becb-456a-8a84-5488f061a777\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"
Aug 04 04:17:25.655144 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.654955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4510463-becb-456a-8a84-5488f061a777-lib-modules\") pod \"perf-node-gather-daemonset-kxt5d\" (UID: \"b4510463-becb-456a-8a84-5488f061a777\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"
Aug 04 04:17:25.655144 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.655031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b4510463-becb-456a-8a84-5488f061a777-proc\") pod \"perf-node-gather-daemonset-kxt5d\" (UID: \"b4510463-becb-456a-8a84-5488f061a777\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"
Aug 04 04:17:25.655144 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.655040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b4510463-becb-456a-8a84-5488f061a777-podres\") pod \"perf-node-gather-daemonset-kxt5d\" (UID: \"b4510463-becb-456a-8a84-5488f061a777\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"
Aug 04 04:17:25.655144 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.655121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj56c\" (UniqueName: \"kubernetes.io/projected/b4510463-becb-456a-8a84-5488f061a777-kube-api-access-zj56c\") pod \"perf-node-gather-daemonset-kxt5d\" (UID: \"b4510463-becb-456a-8a84-5488f061a777\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"
Aug 04 04:17:25.655382 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.655147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b4510463-becb-456a-8a84-5488f061a777-podres\") pod \"perf-node-gather-daemonset-kxt5d\" (UID: \"b4510463-becb-456a-8a84-5488f061a777\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"
Aug 04 04:17:25.655382 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.655153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4510463-becb-456a-8a84-5488f061a777-sys\") pod \"perf-node-gather-daemonset-kxt5d\" (UID: \"b4510463-becb-456a-8a84-5488f061a777\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"
Aug 04 04:17:25.655382 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.655147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4510463-becb-456a-8a84-5488f061a777-lib-modules\") pod \"perf-node-gather-daemonset-kxt5d\" (UID: \"b4510463-becb-456a-8a84-5488f061a777\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"
Aug 04 04:17:25.655382 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.655268    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4510463-becb-456a-8a84-5488f061a777-sys\") pod \"perf-node-gather-daemonset-kxt5d\" (UID: \"b4510463-becb-456a-8a84-5488f061a777\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"
Aug 04 04:17:25.663655 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.663621    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj56c\" (UniqueName: \"kubernetes.io/projected/b4510463-becb-456a-8a84-5488f061a777-kube-api-access-zj56c\") pod \"perf-node-gather-daemonset-kxt5d\" (UID: \"b4510463-becb-456a-8a84-5488f061a777\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"
Aug 04 04:17:25.788267 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.788173    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"
Aug 04 04:17:25.930281 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.930253    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"]
Aug 04 04:17:25.932726 ip-10-0-138-99 kubenswrapper[2579]: W0804 04:17:25.932701    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb4510463_becb_456a_8a84_5488f061a777.slice/crio-f58d3fb3a497772d216381b6c1c4c326903b40c2a24edf8aa2da1c4d7337ce86 WatchSource:0}: Error finding container f58d3fb3a497772d216381b6c1c4c326903b40c2a24edf8aa2da1c4d7337ce86: Status 404 returned error can't find the container with id f58d3fb3a497772d216381b6c1c4c326903b40c2a24edf8aa2da1c4d7337ce86
Aug 04 04:17:25.987348 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:25.987325    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-685448655c-9bx5m_474ea32c-a1b4-42f5-95b6-f8221a087c74/console/0.log"
Aug 04 04:17:26.611072 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:26.611043    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-2k8hf_c3e1258f-b644-4469-b525-eccc09288622/volume-data-source-validator/0.log"
Aug 04 04:17:26.764210 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:26.764166    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d" event={"ID":"b4510463-becb-456a-8a84-5488f061a777","Type":"ContainerStarted","Data":"52483361266a4c81df34f1653a23c391fa8ad575acb4124883ce4de5a360f816"}
Aug 04 04:17:26.764386 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:26.764223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d" event={"ID":"b4510463-becb-456a-8a84-5488f061a777","Type":"ContainerStarted","Data":"f58d3fb3a497772d216381b6c1c4c326903b40c2a24edf8aa2da1c4d7337ce86"}
Aug 04 04:17:26.764386 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:26.764244    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"
Aug 04 04:17:26.781017 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:26.780949    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d" podStartSLOduration=1.7809352280000001 podStartE2EDuration="1.780935228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 04:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:17:26.780123295 +0000 UTC m=+4077.459910431" watchObservedRunningTime="2026-08-04 04:17:26.780935228 +0000 UTC m=+4077.460722346"
Aug 04 04:17:27.757443 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:27.757415    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-np56j_7b43cf1f-077d-485b-8a0d-fdec861aff43/dns/0.log"
Aug 04 04:17:27.779787 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:27.779762    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-np56j_7b43cf1f-077d-485b-8a0d-fdec861aff43/kube-rbac-proxy/0.log"
Aug 04 04:17:27.851680 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:27.851653    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-57qsq_d8bb4fcc-2c06-4bb0-b23b-249bda7fa7b8/dns-node-resolver/0.log"
Aug 04 04:17:28.508770 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:28.508740    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gnp8m_5f76ca57-4490-4583-85a2-95198fb43ffd/node-ca/0.log"
Aug 04 04:17:29.654399 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:29.654372    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69dbcc4dfb-lmwkc_4732185c-0a57-4163-b84b-89c007fbed25/kube-auth-proxy/0.log"
Aug 04 04:17:29.779659 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:29.779629    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-46wnj_2dbe90ed-02ed-428f-ad98-ae60193a9e66/payload-processing/0.log"
Aug 04 04:17:30.652330 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:30.652296    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ztnbv_fc5bd55b-006c-482c-b501-11c7e4a50629/serve-healthcheck-canary/0.log"
Aug 04 04:17:31.300889 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:31.300833    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rzf5k_220034f7-6232-4501-8842-f79fd919fc3c/kube-rbac-proxy/0.log"
Aug 04 04:17:31.323265 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:31.323239    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rzf5k_220034f7-6232-4501-8842-f79fd919fc3c/exporter/0.log"
Aug 04 04:17:31.351382 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:31.351320    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rzf5k_220034f7-6232-4501-8842-f79fd919fc3c/extractor/0.log"
Aug 04 04:17:32.780871 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:32.780844    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-kxt5d"
Aug 04 04:17:33.940976 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:33.940908    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5465dcb4d8-z7qrk_9a292744-1316-4511-bd59-4a1b67a2d714/manager/0.log"
Aug 04 04:17:34.238270 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:34.238237    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx_3c239d70-6373-43eb-acd1-2fb0a5631d82/manager/0.log"
Aug 04 04:17:34.259323 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:34.259293    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fb59c9fb6-xtvrx_3c239d70-6373-43eb-acd1-2fb0a5631d82/copy-manifests/0.log"
Aug 04 04:17:35.766732 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:35.766703    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6488bdc59d-gmpnr_90dafb77-eb17-46d6-9ffa-1e6568c6a439/manager/0.log"
Aug 04 04:17:35.829248 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:35.829220    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69dbcc4dfb-lmwkc_4732185c-0a57-4163-b84b-89c007fbed25/kube-auth-proxy/0.log"
Aug 04 04:17:35.938925 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:35.938842    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-46wnj_2dbe90ed-02ed-428f-ad98-ae60193a9e66/payload-processing/0.log"
Aug 04 04:17:37.512899 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:37.512806    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69dbcc4dfb-lmwkc_4732185c-0a57-4163-b84b-89c007fbed25/kube-auth-proxy/0.log"
Aug 04 04:17:37.623256 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:37.623227    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-46wnj_2dbe90ed-02ed-428f-ad98-ae60193a9e66/payload-processing/0.log"
Aug 04 04:17:40.804339 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:40.804309    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-gzkwr_3c397a61-27ed-4be0-b608-2ab09c04b5a5/migrator/0.log"
Aug 04 04:17:40.832977 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:40.832929    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-gzkwr_3c397a61-27ed-4be0-b608-2ab09c04b5a5/graceful-termination/0.log"
Aug 04 04:17:42.609291 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:42.609262    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nbbch_2c429971-1d7d-4d34-b0fb-c2081a928855/kube-multus-additional-cni-plugins/0.log"
Aug 04 04:17:42.635114 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:42.635085    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nbbch_2c429971-1d7d-4d34-b0fb-c2081a928855/egress-router-binary-copy/0.log"
Aug 04 04:17:42.660283 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:42.660259    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nbbch_2c429971-1d7d-4d34-b0fb-c2081a928855/cni-plugins/0.log"
Aug 04 04:17:42.684882 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:42.684855    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nbbch_2c429971-1d7d-4d34-b0fb-c2081a928855/bond-cni-plugin/0.log"
Aug 04 04:17:42.707945 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:42.707920    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nbbch_2c429971-1d7d-4d34-b0fb-c2081a928855/routeoverride-cni/0.log"
Aug 04 04:17:42.739363 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:42.739342    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nbbch_2c429971-1d7d-4d34-b0fb-c2081a928855/whereabouts-cni-bincopy/0.log"
Aug 04 04:17:42.762089 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:42.762067    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nbbch_2c429971-1d7d-4d34-b0fb-c2081a928855/whereabouts-cni/0.log"
Aug 04 04:17:42.852165 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:42.852133    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-h5jxp_fa970b56-8654-40d5-b836-005f3c50b89f/kube-multus/0.log"
Aug 04 04:17:42.878872 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:42.878794    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-c5d74_b39560be-b9af-4ff7-8a5b-7882c66a03d8/network-metrics-daemon/0.log"
Aug 04 04:17:42.900312 ip-10-0-138-99 kubenswrapper[2579]: I0804 04:17:42.900279    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-c5d74_b39560be-b9af-4ff7-8a5b-7882c66a03d8/kube-rbac-proxy/0.log"