Sep 07 22:31:55.743344 ip-10-0-138-99 systemd[1]: Starting Kubernetes Kubelet...
Sep 07 22:31:56.167502 ip-10-0-138-99 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 22:31:56.167502 ip-10-0-138-99 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 07 22:31:56.167502 ip-10-0-138-99 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 22:31:56.167502 ip-10-0-138-99 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 07 22:31:56.167502 ip-10-0-138-99 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 22:31:56.170409 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.170318    2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 07 22:31:56.176715 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176691    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 22:31:56.176715 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176714    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 22:31:56.176715 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176717    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176721    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176724    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176729    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176732    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176737    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176742    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176745    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176748    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176750    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176753    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176756    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176759    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176762    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176764    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176767    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176769    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176772    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176775    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176778    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 22:31:56.176820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176781    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 22:31:56.177328 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176784    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 22:31:56.177328 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176788    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 22:31:56.177328 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176790    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 22:31:56.177328 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176793    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 22:31:56.177328 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176796    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 22:31:56.177328 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176798    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 22:31:56.177328 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176801    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 22:31:56.177328 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176804    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 22:31:56.177328 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176806    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 22:31:56.177328 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176809    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 22:31:56.177328 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176813    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 22:31:56.177328 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176815    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 22:31:56.177328 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176818    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 22:31:56.177328 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176822    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 22:31:56.177328 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176825    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 22:31:56.177328 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176828    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 22:31:56.177328 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176831    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 22:31:56.177328 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176834    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 22:31:56.177328 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176836    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 22:31:56.177797 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176839    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 22:31:56.177797 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176842    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 22:31:56.177797 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176844    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 22:31:56.177797 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176847    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 22:31:56.177797 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176849    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 22:31:56.177797 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176852    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 22:31:56.177797 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176855    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 22:31:56.177797 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176857    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 22:31:56.177797 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176862    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 22:31:56.177797 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176865    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 22:31:56.177797 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176868    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 22:31:56.177797 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176872    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 22:31:56.177797 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176875    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 22:31:56.177797 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176878    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 22:31:56.177797 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176881    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 22:31:56.177797 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176885    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 22:31:56.177797 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176889    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 22:31:56.177797 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176892    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 22:31:56.177797 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176895    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176898    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176901    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176903    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176906    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176909    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176911    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176914    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176917    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176920    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176922    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176925    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176928    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176930    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176933    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176936    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176939    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176941    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176944    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176946    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 22:31:56.178299 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176949    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176951    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176955    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176957    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176960    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176963    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176966    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176968    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176971    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176974    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176976    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176989    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176992    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.176997    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177000    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177005    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177009    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177012    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177015    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177761    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 22:31:56.178800 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177768    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177771    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177775    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177778    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177782    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177785    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177787    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177790    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177792    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177795    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177797    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177800    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177803    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177806    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177808    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177811    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177814    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177817    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177820    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177823    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 22:31:56.179326 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177826    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 22:31:56.179865 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177829    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 22:31:56.179865 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177831    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 22:31:56.179865 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177834    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 22:31:56.179865 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177837    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 22:31:56.179865 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177840    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 22:31:56.179865 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177842    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 22:31:56.179865 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177845    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 22:31:56.179865 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177848    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 22:31:56.179865 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177850    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 22:31:56.179865 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177853    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 22:31:56.179865 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177855    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 22:31:56.179865 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177858    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 22:31:56.179865 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177860    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 22:31:56.179865 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177863    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 22:31:56.179865 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177866    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 22:31:56.179865 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177868    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 22:31:56.179865 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177871    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 22:31:56.179865 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177874    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 22:31:56.179865 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177876    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177879    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177882    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177885    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177888    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177891    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177894    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177896    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177899    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177901    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177904    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177907    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177909    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177912    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177915    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177919    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177923    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177927    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177930    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177932    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 22:31:56.180355 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177935    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 22:31:56.180857 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177937    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 22:31:56.180857 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177943    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 22:31:56.180857 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177947    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 22:31:56.180857 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177950    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 22:31:56.180857 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177953    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 22:31:56.180857 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177956    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 22:31:56.180857 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177958    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 22:31:56.180857 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177961    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 22:31:56.180857 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177963    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 22:31:56.180857 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177966    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 22:31:56.180857 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177969    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 22:31:56.180857 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177972    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 22:31:56.180857 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177975    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 22:31:56.180857 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177992    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 22:31:56.180857 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.177997    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 22:31:56.180857 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178002    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 22:31:56.180857 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178006    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 22:31:56.180857 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178011    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 22:31:56.180857 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178015    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178019    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178022    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178025    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178028    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178031    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178033    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178036    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178039    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178041    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178044    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178047    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178050    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178053    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178055    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178058    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178060    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178063    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178066    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178068    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 22:31:56.181345 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178469    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178474    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178477    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178480    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178483    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178486    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178489    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178492    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178494    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178497    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178500    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178503    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178505    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178508    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178511    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178514    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178516    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178519    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178522    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178524    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 22:31:56.181837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178527    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178530    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178532    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178535    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178537    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178540    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178543    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178545    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178548    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178550    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178553    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178555    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178558    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178560    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178563    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178565    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178568    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178578    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178581    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178584    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 22:31:56.182351 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178586    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 22:31:56.182843 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178589    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 22:31:56.182843 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178591    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 22:31:56.182843 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178594    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 22:31:56.182843 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178596    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 22:31:56.182843 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178599    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 22:31:56.182843 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178602    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 22:31:56.182843 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178604    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 22:31:56.182843 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178607    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 22:31:56.182843 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178610    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 22:31:56.182843 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178612    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 22:31:56.182843 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178615    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 22:31:56.182843 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178617    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 22:31:56.182843 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178621    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 22:31:56.182843 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178625    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 22:31:56.182843 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178628    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 22:31:56.182843 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178631    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 22:31:56.182843 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178636    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 22:31:56.182843 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178640    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178643    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178646    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178649    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178651    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178654    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178657    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178660    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178663    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178666    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178669    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178671    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178674    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178677    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178682    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178685    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178688    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178691    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178694    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178696    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 22:31:56.183332 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178700    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178703    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178705    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178709    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178711    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178714    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178716    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178719    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178722    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178725    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178727    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178730    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178733    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178735    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178738    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178740    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178743    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178745    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178748    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178750    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 22:31:56.183820 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.178753    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178839    2567 flags.go:64] FLAG: --address="0.0.0.0"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178875    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178882    2567 flags.go:64] FLAG: --anonymous-auth="true"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178887    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178892    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178897    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178901    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178907    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178911    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178914    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178917    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178921    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178924    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178927    2567 flags.go:64] FLAG: --cgroup-root=""
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178930    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178933    2567 flags.go:64] FLAG: --client-ca-file=""
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178936    2567 flags.go:64] FLAG: --cloud-provider="external"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178939    2567 flags.go:64] FLAG: --cluster-dns="[]"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178944    2567 flags.go:64] FLAG: --cluster-domain=""
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178947    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178950    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178953    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178957    2567 flags.go:64] FLAG: --container-log-max-files="5"
Sep 07 22:31:56.184451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178960    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178964    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178967    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178970    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178973    2567 flags.go:64] FLAG: --contention-profiling="false"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178976    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178993    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.178996    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179000    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179005    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179010    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179014    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179019    2567 flags.go:64] FLAG: --enable-load-reader="false"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179023    2567 flags.go:64] FLAG: --enable-server="true"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179029    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179034    2567 flags.go:64] FLAG: --event-burst="100"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179037    2567 flags.go:64] FLAG: --event-qps="50"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179040    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179043    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179046    2567 flags.go:64] FLAG: --eviction-hard=""
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179051    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179054    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179057    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179060    2567 flags.go:64] FLAG: --eviction-soft=""
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179063    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 07 22:31:56.185077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179066    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179069    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179072    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179075    2567 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179078    2567 flags.go:64] FLAG: --fail-swap-on="true"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179081    2567 flags.go:64] FLAG: --feature-gates=""
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179085    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179088    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179091    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179094    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179097    2567 flags.go:64] FLAG: --healthz-port="10248"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179100    2567 flags.go:64] FLAG: --help="false"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179103    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-138-99.ec2.internal"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179107    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179110    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179112    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179116    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179119    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179122    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179125    2567 flags.go:64] FLAG: --image-service-endpoint=""
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179128    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179132    2567 flags.go:64] FLAG: --kube-api-burst="100"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179136    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179139    2567 flags.go:64] FLAG: --kube-api-qps="50"
Sep 07 22:31:56.185703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179142    2567 flags.go:64] FLAG: --kube-reserved=""
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179145    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179147    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179151    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179153    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179156    2567 flags.go:64] FLAG: --lock-file=""
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179159    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179162    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179165    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179171    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179174    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179177    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179179    2567 flags.go:64] FLAG: --logging-format="text"
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179182    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179185    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179189    2567 flags.go:64] FLAG: --manifest-url=""
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179192    2567 flags.go:64] FLAG: --manifest-url-header=""
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179197    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179200    2567 flags.go:64] FLAG: --max-open-files="1000000"
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179204    2567 flags.go:64] FLAG: --max-pods="110"
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179206    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179210    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179212    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179215    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179218    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 07 22:31:56.186321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179221    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179224    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179232    2567 flags.go:64] FLAG: --node-status-max-images="50"
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179235    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179239    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179243    2567 flags.go:64] FLAG: --pod-cidr=""
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179246    2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179252    2567 flags.go:64] FLAG: --pod-manifest-path=""
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179255    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179258    2567 flags.go:64] FLAG: --pods-per-core="0"
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179261    2567 flags.go:64] FLAG: --port="10250"
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179267    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179270    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-035836e7713ce3bcd"
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179273    2567 flags.go:64] FLAG: --qos-reserved=""
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179276    2567 flags.go:64] FLAG: --read-only-port="10255"
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179279    2567 flags.go:64] FLAG: --register-node="true"
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179282    2567 flags.go:64] FLAG: --register-with-taints=""
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179286    2567 flags.go:64] FLAG: --registry-burst="10"
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179288    2567 flags.go:64] FLAG: --registry-qps="5"
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179291    2567 flags.go:64] FLAG: --reserved-cpus=""
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179294    2567 flags.go:64] FLAG: --reserved-memory=""
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179298    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179301    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179304    2567 flags.go:64] FLAG: --rotate-certificates="false"
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179307    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 07 22:31:56.186938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179310    2567 flags.go:64] FLAG: --runonce="false"
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179313    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179316    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179319    2567 flags.go:64] FLAG: --seccomp-default="false"
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179322    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179325    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179328    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179332    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179335    2567 flags.go:64] FLAG: --storage-driver-password="root"
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179338    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179341    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179346    2567 flags.go:64] FLAG: --storage-driver-user="root"
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179349    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179352    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179356    2567 flags.go:64] FLAG: --system-cgroups=""
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179359    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179364    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179367    2567 flags.go:64] FLAG: --tls-cert-file=""
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179371    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179375    2567 flags.go:64] FLAG: --tls-min-version=""
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179378    2567 flags.go:64] FLAG: --tls-private-key-file=""
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179381    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179384    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179387    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179390    2567 flags.go:64] FLAG: --v="2"
Sep 07 22:31:56.187655 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179394    2567 flags.go:64] FLAG: --version="false"
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179398    2567 flags.go:64] FLAG: --vmodule=""
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179402    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.179405    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179522    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179525    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179528    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179531    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179534    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179537    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179539    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179542    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179545    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179547    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179550    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179553    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179557    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179561    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179566    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179569    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 22:31:56.188671 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179572    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179575    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179578    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179581    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179584    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179589    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179592    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179594    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179597    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179599    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179602    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179605    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179608    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179610    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179613    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179615    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179618    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179621    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179624    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179626    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 22:31:56.189402 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179629    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 22:31:56.189949 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179631    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 22:31:56.189949 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179634    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 22:31:56.189949 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179636    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 22:31:56.189949 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179639    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 22:31:56.189949 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179641    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 22:31:56.189949 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179644    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 22:31:56.189949 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179646    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 22:31:56.189949 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179649    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 22:31:56.189949 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179653    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 22:31:56.189949 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179658    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 22:31:56.189949 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179661    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 22:31:56.189949 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179664    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 22:31:56.189949 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179668    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 22:31:56.189949 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179671    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 22:31:56.189949 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179678    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 22:31:56.189949 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179681    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 22:31:56.189949 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179685    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 22:31:56.189949 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179688    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 22:31:56.189949 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179690    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179693    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179695    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179698    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179700    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179703    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179705    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179708    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179710    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179713    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179715    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179718    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179720    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179723    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179726    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179728    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179730    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179733    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179736    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179738    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 22:31:56.190456 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179741    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179743    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179746    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179750    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179752    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179754    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179757    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179760    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179763    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179766    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179771    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179774    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179776    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179779    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179781    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179784    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179786    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179789    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179792    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179794    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 22:31:56.190952 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179797    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 22:31:56.191515 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179799    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 22:31:56.191515 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179802    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 22:31:56.191515 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.179806    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 22:31:56.191515 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.180583    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 22:31:56.191515 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.190956    2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 07 22:31:56.191515 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.190976    2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 07 22:31:56.191515 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.191022    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 07 22:31:56.191515 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.191028    2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 07 22:31:56.191515 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191091    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 22:31:56.191515 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191096    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 22:31:56.191515 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191101    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 22:31:56.191515 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191105    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 22:31:56.191515 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191109    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 22:31:56.191515 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191112    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191115    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191118    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191121    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191124    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191127    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191129    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191132    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191135    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191137    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191140    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191143    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191146    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191149    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191151    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191154    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191157    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191160    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191163    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191166    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 22:31:56.191867 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191169    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 22:31:56.192405 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191172    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 22:31:56.192405 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191175    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 22:31:56.192405 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191178    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 22:31:56.192405 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191180    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 22:31:56.192405 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191183    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 22:31:56.192405 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191186    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 22:31:56.192405 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191188    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 22:31:56.192405 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191192    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 22:31:56.192405 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191195    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 22:31:56.192405 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191197    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 22:31:56.192405 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191200    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 22:31:56.192405 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191202    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 22:31:56.192405 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191205    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 22:31:56.192405 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191209    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 22:31:56.192405 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191213    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 22:31:56.192405 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191216    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 22:31:56.192405 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191218    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 22:31:56.192405 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191221    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 22:31:56.192405 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191224    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191227    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191229    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191232    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191235    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191239    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191243    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191246    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191249    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191251    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191254    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191257    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191260    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191262    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191265    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191268    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191271    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191274    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191276    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191279    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 22:31:56.192875 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191282    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191284    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191287    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191290    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191292    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191295    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191297    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191300    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191303    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191305    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191308    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191310    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191313    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191316    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191318    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191322    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191324    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191327    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191330    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191333    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 22:31:56.193389 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191335    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 22:31:56.193877 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191338    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 22:31:56.193877 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191340    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 22:31:56.193877 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191343    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 22:31:56.193877 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191346    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 22:31:56.193877 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191348    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 22:31:56.193877 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191351    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 22:31:56.193877 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191354    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 22:31:56.193877 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191356    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 22:31:56.193877 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191359    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 22:31:56.193877 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191362    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 22:31:56.193877 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191364    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 22:31:56.193877 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191367    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 22:31:56.193877 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191369    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 22:31:56.193877 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191372    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 22:31:56.194258 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.191377    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 22:31:56.194258 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191498    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 22:31:56.194258 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191504    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 22:31:56.194258 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191506    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 22:31:56.194258 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191509    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 22:31:56.194258 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191512    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 22:31:56.194258 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191515    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 22:31:56.194258 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191519    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 22:31:56.194258 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191521    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 22:31:56.194258 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191524    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 22:31:56.194258 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191527    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 22:31:56.194258 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191530    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 22:31:56.194258 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191533    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 22:31:56.194258 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191536    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191539    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191542    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191545    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191547    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191550    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191553    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191556    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191558    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191561    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191564    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191567    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191569    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191572    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191575    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191577    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191580    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191582    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191585    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191587    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 22:31:56.194617 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191590    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191592    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191595    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191597    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191600    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191603    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191605    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191608    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191610    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191613    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191615    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191618    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191621    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191624    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191627    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191630    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191633    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191637    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191640    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191643    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 22:31:56.195130 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191646    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191649    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191653    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191656    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191659    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191662    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191665    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191667    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191670    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191673    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191675    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191678    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191681    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191684    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191686    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191689    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191691    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191694    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191696    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191699    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 22:31:56.195638 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191701    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 22:31:56.196146 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191704    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 22:31:56.196146 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191706    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 22:31:56.196146 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191710    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 22:31:56.196146 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191714    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 22:31:56.196146 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191716    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 22:31:56.196146 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191720    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 22:31:56.196146 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191723    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 22:31:56.196146 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191726    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 22:31:56.196146 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191729    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 22:31:56.196146 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191731    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 22:31:56.196146 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191734    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 22:31:56.196146 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191737    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 22:31:56.196146 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191739    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 22:31:56.196146 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191742    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 22:31:56.196146 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191744    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 22:31:56.196146 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191747    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 22:31:56.196146 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191750    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 22:31:56.196146 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191752    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 22:31:56.196146 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191755    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 22:31:56.196612 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191757    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 22:31:56.196612 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191760    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 22:31:56.196612 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191763    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 22:31:56.196612 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191765    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 22:31:56.196612 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191768    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 22:31:56.196612 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191771    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 22:31:56.196612 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:56.191773    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 22:31:56.196612 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.191778    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 22:31:56.196612 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.191919    2567 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 07 22:31:56.196612 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.194388    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 07 22:31:56.196612 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.195501    2567 server.go:1019] "Starting client certificate rotation"
Sep 07 22:31:56.196612 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.195624    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 22:31:56.196612 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.195668    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 22:31:56.198426 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.198411    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 22:31:56.201403 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.201367    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 22:31:56.210776 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.210756    2567 log.go:25] "Validated CRI v1 runtime API"
Sep 07 22:31:56.217201 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.217176    2567 log.go:25] "Validated CRI v1 image API"
Sep 07 22:31:56.219479 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.219460    2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 07 22:31:56.222166 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.222139    2567 fs.go:135] Filesystem UUIDs: map[143c6ad7-1486-4375-b149-c8a7d9c2a3bf:/dev/nvme0n1p3 4e72191b-e02c-4ac3-877b-feba3819e76c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 07 22:31:56.222237 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.222167    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 07 22:31:56.226413 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.226397    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 22:31:56.227279 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.227169    2567 manager.go:216] Machine: {Timestamp:2026-09-07 22:31:56.225733031 +0000 UTC m=+0.372829383 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100325 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b9f1254465b74cd22e8ce42a8fd88 SystemUUID:ec2b9f12-5446-5b74-cd22-e8ce42a8fd88 BootID:bd5f1bc7-e9ea-4161-b8da-d33429e1d4ef Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:94:4b:17:ca:e1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:94:4b:17:ca:e1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:24:35:d1:cb:0b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 07 22:31:56.227279 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.227275    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 07 22:31:56.227380 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.227355    2567 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 07 22:31:56.230954 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.230929    2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 07 22:31:56.231106 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.230957    2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-99.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 07 22:31:56.232560 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.232549    2567 topology_manager.go:138] "Creating topology manager with none policy"
Sep 07 22:31:56.232603 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.232562    2567 container_manager_linux.go:309] "Creating device plugin manager"
Sep 07 22:31:56.232603 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.232570    2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 22:31:56.233418 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.233407    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 22:31:56.235286 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.235273    2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 07 22:31:56.240205 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.240189    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 22:31:56.240334 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.240324    2567 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 07 22:31:56.240367 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.240361    2567 kubelet.go:486] "Attempting to sync node with API server"
Sep 07 22:31:56.240401 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.240373    2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 07 22:31:56.240401 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.240393    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 07 22:31:56.240474 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.240402    2567 kubelet.go:398] "Adding apiserver pod source"
Sep 07 22:31:56.240474 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.240412    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 07 22:31:56.240875 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.240842    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 22:31:56.240875 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.240877    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 22:31:56.243275 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.243255    2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 07 22:31:56.246168 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.246144    2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 07 22:31:56.246248 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.246184    2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 07 22:31:56.246371 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.246361    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 07 22:31:56.246405 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.246376    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 07 22:31:56.246405 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.246382    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 07 22:31:56.246405 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.246389    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 07 22:31:56.246405 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.246397    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 07 22:31:56.246405 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.246404    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 07 22:31:56.246972 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.246962    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 07 22:31:56.246972 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.246972    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 07 22:31:56.247046 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.246995    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 07 22:31:56.247046 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.247004    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 07 22:31:56.247046 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.247021    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 07 22:31:56.247046 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.247029    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 07 22:31:56.247944 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.247934    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 07 22:31:56.247944 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.247944    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 07 22:31:56.251890 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.251864    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 07 22:31:56.251890 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:56.251871    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 07 22:31:56.252078 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.251911    2567 server.go:1268] "Started kubelet"
Sep 07 22:31:56.252078 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:56.251939    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 07 22:31:56.252078 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.251999    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 07 22:31:56.252078 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.252044    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 07 22:31:56.252197 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.252095    2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 07 22:31:56.252811 ip-10-0-138-99 systemd[1]: Started Kubernetes Kubelet.
Sep 07 22:31:56.253654 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.253556    2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 07 22:31:56.257093 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.257066    2567 server.go:310] "Adding debug handlers to kubelet server"
Sep 07 22:31:56.257804 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:56.257772    2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 07 22:31:56.261745 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:56.260580    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-99.ec2.internal.18d32a999d32069b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-99.ec2.internal,UID:ip-10-0-138-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-99.ec2.internal,},FirstTimestamp:2026-09-07 22:31:56.251883163 +0000 UTC m=+0.398979520,LastTimestamp:2026-09-07 22:31:56.251883163 +0000 UTC m=+0.398979520,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-99.ec2.internal,}"
Sep 07 22:31:56.261921 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.261898    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 07 22:31:56.262627 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.262611    2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 07 22:31:56.262830 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:56.262811    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Sep 07 22:31:56.263434 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.263417    2567 factory.go:55] Registering systemd factory
Sep 07 22:31:56.263434 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.263435    2567 factory.go:223] Registration of the systemd container factory successfully
Sep 07 22:31:56.263536 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.263458    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 07 22:31:56.263536 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.263459    2567 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 07 22:31:56.263536 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.263492    2567 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 07 22:31:56.263656 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.263633    2567 reconstruct.go:97] "Volume reconstruction finished"
Sep 07 22:31:56.263656 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.263641    2567 reconciler.go:29] "Reconciler: start to sync state"
Sep 07 22:31:56.263724 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.263658    2567 factory.go:153] Registering CRI-O factory
Sep 07 22:31:56.263724 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.263673    2567 factory.go:223] Registration of the crio container factory successfully
Sep 07 22:31:56.263779 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.263731    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 07 22:31:56.263779 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.263755    2567 factory.go:103] Registering Raw factory
Sep 07 22:31:56.263779 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.263769    2567 manager.go:1196] Started watching for new ooms in manager
Sep 07 22:31:56.264129 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.264110    2567 manager.go:349] Starting recovery of all containers
Sep 07 22:31:56.270335 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.270315    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4tzj8"
Sep 07 22:31:56.273392 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:56.273200    2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 07 22:31:56.273563 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:56.273540    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 07 22:31:56.275453 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.275412    2567 manager.go:354] Recovery completed
Sep 07 22:31:56.277370 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:56.277350    2567 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 07 22:31:56.277647 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.277632    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4tzj8"
Sep 07 22:31:56.280762 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.280750    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:31:56.282723 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.282706    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:31:56.282775 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.282740    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:31:56.282775 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.282754    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:31:56.284094 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.284070    2567 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 07 22:31:56.284094 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.284094    2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 07 22:31:56.284230 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.284109    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:31:56.284230 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.284114    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 22:31:56.285277 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.285259    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:31:56.285355 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.285294    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:31:56.285355 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.285305    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:31:56.286338 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.286320    2567 policy_none.go:49] "None policy: Start"
Sep 07 22:31:56.286338 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.286341    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 07 22:31:56.286462 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.286352    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 07 22:31:56.288324 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.288312    2567 policy_none.go:47] "Start"
Sep 07 22:31:56.327680 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.327655    2567 manager.go:337] "Starting Device Plugin manager"
Sep 07 22:31:56.346579 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:56.327703    2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 07 22:31:56.346579 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.327717    2567 server.go:87] "Starting device plugin registration server"
Sep 07 22:31:56.346579 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.328146    2567 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 07 22:31:56.346579 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.328159    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 07 22:31:56.346579 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.328249    2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 07 22:31:56.346579 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.328342    2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 07 22:31:56.346579 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.328353    2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 07 22:31:56.346579 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:56.328945    2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 07 22:31:56.346579 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:56.329024    2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-99.ec2.internal\" not found"
Sep 07 22:31:56.367285 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.367251    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 07 22:31:56.368492 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.368473    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 07 22:31:56.368604 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.368520    2567 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 07 22:31:56.368604 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.368543    2567 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 07 22:31:56.368694 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:56.368632    2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 07 22:31:56.370724 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.370706    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 22:31:56.428409 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.428294    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:31:56.429327 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.429310    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:31:56.429386 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.429350    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:31:56.429386 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.429364    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:31:56.429463 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.429390    2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-99.ec2.internal"
Sep 07 22:31:56.437974 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.437954    2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-99.ec2.internal"
Sep 07 22:31:56.438050 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:56.438003    2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-99.ec2.internal\": node \"ip-10-0-138-99.ec2.internal\" not found"
Sep 07 22:31:56.460342 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:56.460309    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Sep 07 22:31:56.469661 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.469618    2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal"]
Sep 07 22:31:56.469750 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.469719    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:31:56.471272 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.471253    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:31:56.471371 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.471283    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:31:56.471371 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.471293    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:31:56.473478 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.473467    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:31:56.473642 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.473625    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal"
Sep 07 22:31:56.473688 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.473660    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:31:56.474213 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.474199    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:31:56.474274 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.474223    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:31:56.474274 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.474233    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:31:56.474346 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.474201    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:31:56.474346 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.474303    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:31:56.474346 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.474312    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:31:56.476468 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.476452    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal"
Sep 07 22:31:56.476581 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.476475    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:31:56.477176 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.477159    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:31:56.477257 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.477191    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:31:56.477257 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.477206    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:31:56.505800 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:56.505776    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-99.ec2.internal\" not found" node="ip-10-0-138-99.ec2.internal"
Sep 07 22:31:56.510297 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:56.510275    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-99.ec2.internal\" not found" node="ip-10-0-138-99.ec2.internal"
Sep 07 22:31:56.560802 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:56.560764    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Sep 07 22:31:56.565087 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.565066    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9c0cba75e87ce3cbaf5728c72e95a717-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal\" (UID: \"9c0cba75e87ce3cbaf5728c72e95a717\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal"
Sep 07 22:31:56.565173 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.565095    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c0cba75e87ce3cbaf5728c72e95a717-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal\" (UID: \"9c0cba75e87ce3cbaf5728c72e95a717\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal"
Sep 07 22:31:56.565173 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:56.565112    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c67f07165e3fd7605dfe27e9a9c951a-config\") pod \"kube-apiserver-proxy-ip-10-0-138-99.ec2.internal\" (UID: \"9c67f07165e3fd7605dfe27e9a9c951a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal"
Sep 07 22:31:57.751534 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:57.751501    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Sep 07 22:31:57.755887 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:57.755865    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9c0cba75e87ce3cbaf5728c72e95a717-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal\" (UID: \"9c0cba75e87ce3cbaf5728c72e95a717\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal"
Sep 07 22:31:57.755945 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:57.755895    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c0cba75e87ce3cbaf5728c72e95a717-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal\" (UID: \"9c0cba75e87ce3cbaf5728c72e95a717\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal"
Sep 07 22:31:57.755945 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:57.755915    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c67f07165e3fd7605dfe27e9a9c951a-config\") pod \"kube-apiserver-proxy-ip-10-0-138-99.ec2.internal\" (UID: \"9c67f07165e3fd7605dfe27e9a9c951a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal"
Sep 07 22:31:57.756007 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:57.755980    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9c0cba75e87ce3cbaf5728c72e95a717-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal\" (UID: \"9c0cba75e87ce3cbaf5728c72e95a717\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal"
Sep 07 22:31:57.756007 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:57.755990    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c0cba75e87ce3cbaf5728c72e95a717-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal\" (UID: \"9c0cba75e87ce3cbaf5728c72e95a717\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal"
Sep 07 22:31:57.756069 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:57.756036    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c67f07165e3fd7605dfe27e9a9c951a-config\") pod \"kube-apiserver-proxy-ip-10-0-138-99.ec2.internal\" (UID: \"9c67f07165e3fd7605dfe27e9a9c951a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal"
Sep 07 22:31:57.852330 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:57.852297    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Sep 07 22:31:57.898784 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:57.898744    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal"
Sep 07 22:31:57.903564 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:57.903546    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal"
Sep 07 22:31:57.952665 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:57.952631    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Sep 07 22:31:58.053174 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:58.053142    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Sep 07 22:31:58.153740 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:58.153657    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Sep 07 22:31:58.254143 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:58.254105    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Sep 07 22:31:58.285455 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:58.285438    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 07 22:31:58.285866 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:58.285575    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 07 22:31:58.354332 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:58.354295    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Sep 07 22:31:58.371107 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:58.371073    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-06 22:26:57 +0000 UTC" deadline="2028-05-04 17:43:23.717135881 +0000 UTC"
Sep 07 22:31:58.371107 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:58.371104    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14515h11m25.346034876s"
Sep 07 22:31:58.454609 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:58.454526    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-99.ec2.internal\" not found"
Sep 07 22:31:58.536968 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:58.536930    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 22:31:58.553583 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:58.553545    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal"
Sep 07 22:31:58.565185 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:58.565163    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 22:31:58.566695 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:58.566674    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal"
Sep 07 22:31:58.572038 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:58.572006    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 22:31:58.607510 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:58.607476    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0cba75e87ce3cbaf5728c72e95a717.slice/crio-b5aa10ca98715eac6e3ca7d6afa7abd2586389484e647092d69a151fac00a619 WatchSource:0}: Error finding container b5aa10ca98715eac6e3ca7d6afa7abd2586389484e647092d69a151fac00a619: Status 404 returned error can't find the container with id b5aa10ca98715eac6e3ca7d6afa7abd2586389484e647092d69a151fac00a619
Sep 07 22:31:58.607764 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:31:58.607739    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c67f07165e3fd7605dfe27e9a9c951a.slice/crio-ffeabb7eb1fe99cf3809573e7fc008d68ee7a924ee3486cd7382d9675d8d8c73 WatchSource:0}: Error finding container ffeabb7eb1fe99cf3809573e7fc008d68ee7a924ee3486cd7382d9675d8d8c73: Status 404 returned error can't find the container with id ffeabb7eb1fe99cf3809573e7fc008d68ee7a924ee3486cd7382d9675d8d8c73
Sep 07 22:31:58.612005 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:58.611985    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 22:31:58.760780 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:58.760571    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 22:31:58.876119 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:58.876090    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 22:31:59.236852 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.236820    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 22:31:59.332763 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.332731    2567 apiserver.go:52] "Watching apiserver"
Sep 07 22:31:59.338430 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.338394    2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 07 22:31:59.340844 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.340808    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-n6df9","kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv","openshift-image-registry/node-ca-cz944","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal","openshift-multus/multus-b8xx6","openshift-multus/network-metrics-daemon-wrtvx","openshift-network-diagnostics/network-check-target-q6zr5","kube-system/global-pull-secret-syncer-jd8ww","openshift-cluster-node-tuning-operator/tuned-kf2z2","openshift-dns/node-resolver-vb75n","openshift-multus/multus-additional-cni-plugins-x7bn4","openshift-network-operator/iptables-alerter-2mmwh","openshift-ovn-kubernetes/ovnkube-node-4fnj7"]
Sep 07 22:31:59.343631 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.343358    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:31:59.343631 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:59.343456    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wrtvx" podUID="f95816cb-c07f-4e9f-b828-930dcf1c06eb"
Sep 07 22:31:59.345683 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.345661    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vb75n"
Sep 07 22:31:59.347739 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.347688    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 07 22:31:59.348047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.348025    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r769h\""
Sep 07 22:31:59.348291 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.348272    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 07 22:31:59.348817 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.348798    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.350281 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.350261    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cz944"
Sep 07 22:31:59.351108 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.351089    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-grvkj\""
Sep 07 22:31:59.351343 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.351319    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 07 22:31:59.351619 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.351599    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 07 22:31:59.351807 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.351790    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 07 22:31:59.352164 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.352143    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 07 22:31:59.352344 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.352321    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 07 22:31:59.352566 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.352549    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fw4kc\""
Sep 07 22:31:59.352880 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.352852    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 07 22:31:59.353200 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.353181    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 07 22:31:59.354212 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.354191    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.355001 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.354978    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.356688 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.356666    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 07 22:31:59.357583 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.357112    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 07 22:31:59.358086 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.357798    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 07 22:31:59.358171 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.358147    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 07 22:31:59.359861 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.359438    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 07 22:31:59.359861 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.359800    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mb2sz\""
Sep 07 22:31:59.359861 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.359845    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 07 22:31:59.360057 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.359976    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 07 22:31:59.360113 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.360075    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 07 22:31:59.360283 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.360181    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lclj8\""
Sep 07 22:31:59.360667 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.360378    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 07 22:31:59.362357 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.360860    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2mmwh"
Sep 07 22:31:59.362357 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.360995    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:31:59.362357 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:59.361050    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jd8ww" podUID="1e97b2da-120f-4a72-a6ea-907bcb72a13e"
Sep 07 22:31:59.362357 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.361306    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 07 22:31:59.363001 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.362826    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:31:59.363001 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:59.362888    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6zr5" podUID="0a7d1434-99e1-4ab9-8b17-08aebf32c93a"
Sep 07 22:31:59.363135 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363065    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56a27e39-3141-4765-8e18-d8195a6c567e-host\") pod \"node-ca-cz944\" (UID: \"56a27e39-3141-4765-8e18-d8195a6c567e\") " pod="openshift-image-registry/node-ca-cz944"
Sep 07 22:31:59.363135 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363095    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-var-lib-openvswitch\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.363135 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363120    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f2dfdc2-be64-43c6-813f-f34f09577b9b-ovn-node-metrics-cert\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.363341 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363144    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-multus-socket-dir-parent\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.363341 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363168    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-host-var-lib-cni-bin\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.363341 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363190    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgzgl\" (UniqueName: \"kubernetes.io/projected/afca2585-5d1b-4577-8efc-882374f40887-kube-api-access-pgzgl\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.363341 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363223    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzmrh\" (UniqueName: \"kubernetes.io/projected/b09b22bc-14fb-423b-b7b4-0d3620da9bca-kube-api-access-mzmrh\") pod \"node-resolver-vb75n\" (UID: \"b09b22bc-14fb-423b-b7b4-0d3620da9bca\") " pod="openshift-dns/node-resolver-vb75n"
Sep 07 22:31:59.363341 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363259    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c9d878f7-d905-4ad5-a824-bdf4a0f25476-etc-selinux\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.363341 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363286    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-kubelet\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.363341 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363310    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f2dfdc2-be64-43c6-813f-f34f09577b9b-env-overrides\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.363341 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363338    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-cnibin\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.364977 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363361    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-host-run-k8s-cni-cncf-io\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.364977 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363384    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b09b22bc-14fb-423b-b7b4-0d3620da9bca-hosts-file\") pod \"node-resolver-vb75n\" (UID: \"b09b22bc-14fb-423b-b7b4-0d3620da9bca\") " pod="openshift-dns/node-resolver-vb75n"
Sep 07 22:31:59.364977 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363410    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-node-log\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.364977 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363432    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-cni-netd\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.364977 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363457    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.364977 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363762    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-multus-conf-dir\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.364977 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363817    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c9d878f7-d905-4ad5-a824-bdf4a0f25476-sys-fs\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.364977 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363851    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-strdk\" (UniqueName: \"kubernetes.io/projected/c9d878f7-d905-4ad5-a824-bdf4a0f25476-kube-api-access-strdk\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.364977 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.363883    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw2wc\" (UniqueName: \"kubernetes.io/projected/56a27e39-3141-4765-8e18-d8195a6c567e-kube-api-access-vw2wc\") pod \"node-ca-cz944\" (UID: \"56a27e39-3141-4765-8e18-d8195a6c567e\") " pod="openshift-image-registry/node-ca-cz944"
Sep 07 22:31:59.364977 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.364065    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-run-ovn\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.364977 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.364092    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-system-cni-dir\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.364977 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.364122    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afca2585-5d1b-4577-8efc-882374f40887-cni-binary-copy\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.364977 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.364173    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-host-run-netns\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.364977 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.364204    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-slash\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.364977 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.364421    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-etc-openvswitch\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.364977 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.364459    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-log-socket\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.365642 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.364489    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-cni-bin\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.365642 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.365037    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.367032 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.365767    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-host-var-lib-cni-multus\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.367032 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.365815    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-host-var-lib-kubelet\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.367032 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.365845    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/afca2585-5d1b-4577-8efc-882374f40887-multus-daemon-config\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.367032 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.365879    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-host-run-multus-certs\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.367032 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.365915    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-run-systemd\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.367032 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.365980    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f2dfdc2-be64-43c6-813f-f34f09577b9b-ovnkube-config\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.367032 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.366025    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f2dfdc2-be64-43c6-813f-f34f09577b9b-ovnkube-script-lib\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.367032 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.366054    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-multus-cni-dir\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.367032 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.366071    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-etc-kubernetes\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.367032 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.366090    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs\") pod \"network-metrics-daemon-wrtvx\" (UID: \"f95816cb-c07f-4e9f-b828-930dcf1c06eb\") " pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:31:59.367032 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.366110    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b09b22bc-14fb-423b-b7b4-0d3620da9bca-tmp-dir\") pod \"node-resolver-vb75n\" (UID: \"b09b22bc-14fb-423b-b7b4-0d3620da9bca\") " pod="openshift-dns/node-resolver-vb75n"
Sep 07 22:31:59.367032 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.366139    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-systemd-units\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.367032 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.366174    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s57fk\" (UniqueName: \"kubernetes.io/projected/0f2dfdc2-be64-43c6-813f-f34f09577b9b-kube-api-access-s57fk\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.367032 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.366232    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsnb5\" (UniqueName: \"kubernetes.io/projected/f95816cb-c07f-4e9f-b828-930dcf1c06eb-kube-api-access-qsnb5\") pod \"network-metrics-daemon-wrtvx\" (UID: \"f95816cb-c07f-4e9f-b828-930dcf1c06eb\") " pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:31:59.367032 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.366303    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c9d878f7-d905-4ad5-a824-bdf4a0f25476-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.367032 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.366338    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56a27e39-3141-4765-8e18-d8195a6c567e-serviceca\") pod \"node-ca-cz944\" (UID: \"56a27e39-3141-4765-8e18-d8195a6c567e\") " pod="openshift-image-registry/node-ca-cz944"
Sep 07 22:31:59.367032 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.366370    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-run-netns\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.367952 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.366403    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-run-openvswitch\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.367952 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.366434    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.367952 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.366461    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-hostroot\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.367952 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.366504    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c9d878f7-d905-4ad5-a824-bdf4a0f25476-socket-dir\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.367952 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.367541    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c9d878f7-d905-4ad5-a824-bdf4a0f25476-registration-dir\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.367952 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.367583    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c9d878f7-d905-4ad5-a824-bdf4a0f25476-device-dir\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.367952 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.367808    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 07 22:31:59.368236 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.368066    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 07 22:31:59.368302 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.368284    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 07 22:31:59.368623 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.368602    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pjslb\""
Sep 07 22:31:59.368760 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.368740    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 07 22:31:59.368848 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.368781    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w9cms\""
Sep 07 22:31:59.368905 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.368893    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 07 22:31:59.370368 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.370346    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 22:31:59.374312 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.374282    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n6df9"
Sep 07 22:31:59.376335 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.376313    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.377103 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.377084    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 07 22:31:59.377211 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.377150    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-q92qn\""
Sep 07 22:31:59.377289 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.377083    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 07 22:31:59.378778 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.378758    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 07 22:31:59.378971 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.378956    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 07 22:31:59.379410 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.379392    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7zrdn\""
Sep 07 22:31:59.382780 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.382755    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qkk7b"
Sep 07 22:31:59.392592 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.392554    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qkk7b"
Sep 07 22:31:59.455162 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.455134    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 07 22:31:59.463425 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.463351    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal" event={"ID":"9c0cba75e87ce3cbaf5728c72e95a717","Type":"ContainerStarted","Data":"b5aa10ca98715eac6e3ca7d6afa7abd2586389484e647092d69a151fac00a619"}
Sep 07 22:31:59.464545 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.464513    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal" event={"ID":"9c67f07165e3fd7605dfe27e9a9c951a","Type":"ContainerStarted","Data":"ffeabb7eb1fe99cf3809573e7fc008d68ee7a924ee3486cd7382d9675d8d8c73"}
Sep 07 22:31:59.467864 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.467835    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c9d878f7-d905-4ad5-a824-bdf4a0f25476-sys-fs\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.467984 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.467873    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw2wc\" (UniqueName: \"kubernetes.io/projected/56a27e39-3141-4765-8e18-d8195a6c567e-kube-api-access-vw2wc\") pod \"node-ca-cz944\" (UID: \"56a27e39-3141-4765-8e18-d8195a6c567e\") " pod="openshift-image-registry/node-ca-cz944"
Sep 07 22:31:59.467984 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.467907    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-etc-sysctl-conf\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.467984 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.467931    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-run\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.467984 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.467954    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.467984 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.467973    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-slash\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.468225 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.467988    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/afca2585-5d1b-4577-8efc-882374f40887-multus-daemon-config\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.468225 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468004    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-etc-kubernetes\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.468225 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468019    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c9d878f7-d905-4ad5-a824-bdf4a0f25476-socket-dir\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.468225 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468032    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-etc-systemd\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.468225 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468050    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.468225 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468070    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-run-systemd\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.468225 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468088    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f2dfdc2-be64-43c6-813f-f34f09577b9b-ovnkube-config\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.468225 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468112    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-multus-cni-dir\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.468225 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468136    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs\") pod \"network-metrics-daemon-wrtvx\" (UID: \"f95816cb-c07f-4e9f-b828-930dcf1c06eb\") " pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:31:59.468225 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468161    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b09b22bc-14fb-423b-b7b4-0d3620da9bca-tmp-dir\") pod \"node-resolver-vb75n\" (UID: \"b09b22bc-14fb-423b-b7b4-0d3620da9bca\") " pod="openshift-dns/node-resolver-vb75n"
Sep 07 22:31:59.468643 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468234    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c9d878f7-d905-4ad5-a824-bdf4a0f25476-sys-fs\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.468643 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468300    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-etc-sysctl-d\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.468643 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468332    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e97b2da-120f-4a72-a6ea-907bcb72a13e-dbus\") pod \"global-pull-secret-syncer-jd8ww\" (UID: \"1e97b2da-120f-4a72-a6ea-907bcb72a13e\") " pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:31:59.468643 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468356    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.468643 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468390    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-systemd-units\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.468643 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468414    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s57fk\" (UniqueName: \"kubernetes.io/projected/0f2dfdc2-be64-43c6-813f-f34f09577b9b-kube-api-access-s57fk\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.468643 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468448    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56a27e39-3141-4765-8e18-d8195a6c567e-serviceca\") pod \"node-ca-cz944\" (UID: \"56a27e39-3141-4765-8e18-d8195a6c567e\") " pod="openshift-image-registry/node-ca-cz944"
Sep 07 22:31:59.468643 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468472    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhs42\" (UniqueName: \"kubernetes.io/projected/11688621-db80-4166-80e2-b07c9755e04a-kube-api-access-lhs42\") pod \"iptables-alerter-2mmwh\" (UID: \"11688621-db80-4166-80e2-b07c9755e04a\") " pod="openshift-network-operator/iptables-alerter-2mmwh"
Sep 07 22:31:59.468643 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468498    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-run-openvswitch\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.468643 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468509    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b09b22bc-14fb-423b-b7b4-0d3620da9bca-tmp-dir\") pod \"node-resolver-vb75n\" (UID: \"b09b22bc-14fb-423b-b7b4-0d3620da9bca\") " pod="openshift-dns/node-resolver-vb75n"
Sep 07 22:31:59.468643 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468541    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c9d878f7-d905-4ad5-a824-bdf4a0f25476-device-dir\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.468643 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468566    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-var-lib-openvswitch\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.468643 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468593    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-etc-kubernetes\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.468643 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468590    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f2dfdc2-be64-43c6-813f-f34f09577b9b-ovn-node-metrics-cert\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.469255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468705    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c9d878f7-d905-4ad5-a824-bdf4a0f25476-socket-dir\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.469255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468739    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-multus-socket-dir-parent\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.469255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468766    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgzgl\" (UniqueName: \"kubernetes.io/projected/afca2585-5d1b-4577-8efc-882374f40887-kube-api-access-pgzgl\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.469255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468764    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/afca2585-5d1b-4577-8efc-882374f40887-multus-daemon-config\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.469255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468808    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f2dfdc2-be64-43c6-813f-f34f09577b9b-env-overrides\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.469255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468840    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-host-run-k8s-cni-cncf-io\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.469255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468865    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-node-log\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.469255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468907    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-cni-netd\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.469255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468936    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.469255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468964    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-multus-conf-dir\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.469255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468991    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-strdk\" (UniqueName: \"kubernetes.io/projected/c9d878f7-d905-4ad5-a824-bdf4a0f25476-kube-api-access-strdk\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.469255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469021    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-cnibin\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.469255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469006    2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 07 22:31:59.469255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469049    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-run-ovn\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.469255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469073    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-system-cni-dir\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.469255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469102    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afca2585-5d1b-4577-8efc-882374f40887-cni-binary-copy\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.469255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469119    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-multus-cni-dir\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.469255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469126    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-host-run-netns\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.470047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469163    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-var-lib-kubelet\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.470047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469191    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-host-run-netns\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.470047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469193    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-host\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.470047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469224    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e97b2da-120f-4a72-a6ea-907bcb72a13e-kubelet-config\") pod \"global-pull-secret-syncer-jd8ww\" (UID: \"1e97b2da-120f-4a72-a6ea-907bcb72a13e\") " pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:31:59.470047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469274    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f2dfdc2-be64-43c6-813f-f34f09577b9b-ovnkube-config\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.470047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469358    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-multus-socket-dir-parent\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.470047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469400    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-run-openvswitch\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.470047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.468936    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-run-systemd\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.470047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469488    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-slash\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.470047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469490    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-host-run-k8s-cni-cncf-io\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.470047 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:59.469559    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:31:59.470047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469577    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c9d878f7-d905-4ad5-a824-bdf4a0f25476-device-dir\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.470047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469581    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-node-log\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.470047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469613    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-var-lib-openvswitch\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.470047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469618    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-multus-conf-dir\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.470047 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:59.469661    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs podName:f95816cb-c07f-4e9f-b828-930dcf1c06eb nodeName:}" failed. No retries permitted until 2026-09-07 22:31:59.969641327 +0000 UTC m=+3.026474284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs") pod "network-metrics-daemon-wrtvx" (UID: "f95816cb-c07f-4e9f-b828-930dcf1c06eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:31:59.470047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469657    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.470738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469660    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-cni-netd\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.470738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469743    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-system-cni-dir\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.470738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469769    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-systemd-units\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.470738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469783    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56a27e39-3141-4765-8e18-d8195a6c567e-serviceca\") pod \"node-ca-cz944\" (UID: \"56a27e39-3141-4765-8e18-d8195a6c567e\") " pod="openshift-image-registry/node-ca-cz944"
Sep 07 22:31:59.470738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469797    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-run-ovn\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.470738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469883    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-etc-openvswitch\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.470738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469922    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-log-socket\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.470738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469947    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-cni-bin\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.470738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469959    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-etc-openvswitch\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.470738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469982    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-host-var-lib-cni-multus\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.470738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.469996    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-log-socket\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.470738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470020    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-host-var-lib-kubelet\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.470738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470028    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-cni-bin\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.470738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470046    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-host-run-multus-certs\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.470738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470050    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-host-var-lib-cni-multus\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.470738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470069    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56a27e39-3141-4765-8e18-d8195a6c567e-host\") pod \"node-ca-cz944\" (UID: \"56a27e39-3141-4765-8e18-d8195a6c567e\") " pod="openshift-image-registry/node-ca-cz944"
Sep 07 22:31:59.470738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470097    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-host-run-multus-certs\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.470738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470110    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-host-var-lib-kubelet\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.471569 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470111    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-lib-modules\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.471569 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470135    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56a27e39-3141-4765-8e18-d8195a6c567e-host\") pod \"node-ca-cz944\" (UID: \"56a27e39-3141-4765-8e18-d8195a6c567e\") " pod="openshift-image-registry/node-ca-cz944"
Sep 07 22:31:59.471569 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470153    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f2dfdc2-be64-43c6-813f-f34f09577b9b-ovnkube-script-lib\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.471569 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470177    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c9d878f7-d905-4ad5-a824-bdf4a0f25476-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.471569 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470205    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11688621-db80-4166-80e2-b07c9755e04a-iptables-alerter-script\") pod \"iptables-alerter-2mmwh\" (UID: \"11688621-db80-4166-80e2-b07c9755e04a\") " pod="openshift-network-operator/iptables-alerter-2mmwh"
Sep 07 22:31:59.471569 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470266    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c9d878f7-d905-4ad5-a824-bdf4a0f25476-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.471569 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470426    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f2dfdc2-be64-43c6-813f-f34f09577b9b-env-overrides\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.471569 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470660    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afca2585-5d1b-4577-8efc-882374f40887-cni-binary-copy\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.471569 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470742    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-etc-sysconfig\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.471569 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470776    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c64443d-b524-482c-99f9-3719f2a401c7-tmp\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.471569 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470811    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tznsr\" (UniqueName: \"kubernetes.io/projected/7c64443d-b524-482c-99f9-3719f2a401c7-kube-api-access-tznsr\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.471569 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470837    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret\") pod \"global-pull-secret-syncer-jd8ww\" (UID: \"1e97b2da-120f-4a72-a6ea-907bcb72a13e\") " pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:31:59.471569 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470856    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f2dfdc2-be64-43c6-813f-f34f09577b9b-ovnkube-script-lib\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.471569 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470866    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsnb5\" (UniqueName: \"kubernetes.io/projected/f95816cb-c07f-4e9f-b828-930dcf1c06eb-kube-api-access-qsnb5\") pod \"network-metrics-daemon-wrtvx\" (UID: \"f95816cb-c07f-4e9f-b828-930dcf1c06eb\") " pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:31:59.471569 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470887    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c64443d-b524-482c-99f9-3719f2a401c7-etc-tuned\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.471569 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470911    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-run-netns\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.471569 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470941    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.472379 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470964    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-hostroot\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.472379 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.470986    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c9d878f7-d905-4ad5-a824-bdf4a0f25476-registration-dir\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.472379 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471007    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-etc-modprobe-d\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.472379 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471016    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.472379 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471028    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5efbc01b-b5b9-40df-9ba9-71acdfd8be6e-konnectivity-ca\") pod \"konnectivity-agent-n6df9\" (UID: \"5efbc01b-b5b9-40df-9ba9-71acdfd8be6e\") " pod="kube-system/konnectivity-agent-n6df9"
Sep 07 22:31:59.472379 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471087    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7v6m\" (UniqueName: \"kubernetes.io/projected/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-kube-api-access-c7v6m\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.472379 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471097    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c9d878f7-d905-4ad5-a824-bdf4a0f25476-registration-dir\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.472379 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471114    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-host-var-lib-cni-bin\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.472379 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471129    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-hostroot\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.472379 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471061    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-run-netns\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.472379 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471151    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzmrh\" (UniqueName: \"kubernetes.io/projected/b09b22bc-14fb-423b-b7b4-0d3620da9bca-kube-api-access-mzmrh\") pod \"node-resolver-vb75n\" (UID: \"b09b22bc-14fb-423b-b7b4-0d3620da9bca\") " pod="openshift-dns/node-resolver-vb75n"
Sep 07 22:31:59.472379 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471165    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-host-var-lib-cni-bin\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.472379 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471174    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c9d878f7-d905-4ad5-a824-bdf4a0f25476-etc-selinux\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.472379 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471197    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11688621-db80-4166-80e2-b07c9755e04a-host-slash\") pod \"iptables-alerter-2mmwh\" (UID: \"11688621-db80-4166-80e2-b07c9755e04a\") " pod="openshift-network-operator/iptables-alerter-2mmwh"
Sep 07 22:31:59.472379 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471220    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz2t8\" (UniqueName: \"kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8\") pod \"network-check-target-q6zr5\" (UID: \"0a7d1434-99e1-4ab9-8b17-08aebf32c93a\") " pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:31:59.472379 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471258    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-kubelet\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.472379 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471266    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c9d878f7-d905-4ad5-a824-bdf4a0f25476-etc-selinux\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.473198 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471287    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-cnibin\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.473198 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471310    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b09b22bc-14fb-423b-b7b4-0d3620da9bca-hosts-file\") pod \"node-resolver-vb75n\" (UID: \"b09b22bc-14fb-423b-b7b4-0d3620da9bca\") " pod="openshift-dns/node-resolver-vb75n"
Sep 07 22:31:59.473198 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471332    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f2dfdc2-be64-43c6-813f-f34f09577b9b-host-kubelet\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.473198 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471333    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-etc-kubernetes\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.473198 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471369    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-sys\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.473198 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471380    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afca2585-5d1b-4577-8efc-882374f40887-cnibin\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.473198 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471417    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b09b22bc-14fb-423b-b7b4-0d3620da9bca-hosts-file\") pod \"node-resolver-vb75n\" (UID: \"b09b22bc-14fb-423b-b7b4-0d3620da9bca\") " pod="openshift-dns/node-resolver-vb75n"
Sep 07 22:31:59.473198 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471409    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5efbc01b-b5b9-40df-9ba9-71acdfd8be6e-agent-certs\") pod \"konnectivity-agent-n6df9\" (UID: \"5efbc01b-b5b9-40df-9ba9-71acdfd8be6e\") " pod="kube-system/konnectivity-agent-n6df9"
Sep 07 22:31:59.473198 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471452    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-system-cni-dir\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.473198 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.471480    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.473908 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.473886    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f2dfdc2-be64-43c6-813f-f34f09577b9b-ovn-node-metrics-cert\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.486347 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.486216    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-strdk\" (UniqueName: \"kubernetes.io/projected/c9d878f7-d905-4ad5-a824-bdf4a0f25476-kube-api-access-strdk\") pod \"aws-ebs-csi-driver-node-p5vqv\" (UID: \"c9d878f7-d905-4ad5-a824-bdf4a0f25476\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.486347 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.486224    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsnb5\" (UniqueName: \"kubernetes.io/projected/f95816cb-c07f-4e9f-b828-930dcf1c06eb-kube-api-access-qsnb5\") pod \"network-metrics-daemon-wrtvx\" (UID: \"f95816cb-c07f-4e9f-b828-930dcf1c06eb\") " pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:31:59.487370 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.486600    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw2wc\" (UniqueName: \"kubernetes.io/projected/56a27e39-3141-4765-8e18-d8195a6c567e-kube-api-access-vw2wc\") pod \"node-ca-cz944\" (UID: \"56a27e39-3141-4765-8e18-d8195a6c567e\") " pod="openshift-image-registry/node-ca-cz944"
Sep 07 22:31:59.492650 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.492626    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzmrh\" (UniqueName: \"kubernetes.io/projected/b09b22bc-14fb-423b-b7b4-0d3620da9bca-kube-api-access-mzmrh\") pod \"node-resolver-vb75n\" (UID: \"b09b22bc-14fb-423b-b7b4-0d3620da9bca\") " pod="openshift-dns/node-resolver-vb75n"
Sep 07 22:31:59.493511 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.493484    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s57fk\" (UniqueName: \"kubernetes.io/projected/0f2dfdc2-be64-43c6-813f-f34f09577b9b-kube-api-access-s57fk\") pod \"ovnkube-node-4fnj7\" (UID: \"0f2dfdc2-be64-43c6-813f-f34f09577b9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.494923 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.494899    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgzgl\" (UniqueName: \"kubernetes.io/projected/afca2585-5d1b-4577-8efc-882374f40887-kube-api-access-pgzgl\") pod \"multus-b8xx6\" (UID: \"afca2585-5d1b-4577-8efc-882374f40887\") " pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.572539 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.572500    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-etc-modprobe-d\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.572712 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.572548    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5efbc01b-b5b9-40df-9ba9-71acdfd8be6e-konnectivity-ca\") pod \"konnectivity-agent-n6df9\" (UID: \"5efbc01b-b5b9-40df-9ba9-71acdfd8be6e\") " pod="kube-system/konnectivity-agent-n6df9"
Sep 07 22:31:59.572712 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.572687    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7v6m\" (UniqueName: \"kubernetes.io/projected/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-kube-api-access-c7v6m\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.572821 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.572742    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11688621-db80-4166-80e2-b07c9755e04a-host-slash\") pod \"iptables-alerter-2mmwh\" (UID: \"11688621-db80-4166-80e2-b07c9755e04a\") " pod="openshift-network-operator/iptables-alerter-2mmwh"
Sep 07 22:31:59.572821 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.572770    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz2t8\" (UniqueName: \"kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8\") pod \"network-check-target-q6zr5\" (UID: \"0a7d1434-99e1-4ab9-8b17-08aebf32c93a\") " pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:31:59.572821 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.572692    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-etc-modprobe-d\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.572821 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.572797    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-etc-kubernetes\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.573003 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.572820    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-sys\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.573003 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.572843    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5efbc01b-b5b9-40df-9ba9-71acdfd8be6e-agent-certs\") pod \"konnectivity-agent-n6df9\" (UID: \"5efbc01b-b5b9-40df-9ba9-71acdfd8be6e\") " pod="kube-system/konnectivity-agent-n6df9"
Sep 07 22:31:59.573003 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.572866    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-system-cni-dir\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.573003 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.572892    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.573003 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.572916    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-etc-sysctl-conf\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.573003 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.572937    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-run\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.573003 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.572960    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.573003 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.572989    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-etc-systemd\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.573448 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573003    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11688621-db80-4166-80e2-b07c9755e04a-host-slash\") pod \"iptables-alerter-2mmwh\" (UID: \"11688621-db80-4166-80e2-b07c9755e04a\") " pod="openshift-network-operator/iptables-alerter-2mmwh"
Sep 07 22:31:59.573448 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573012    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.573448 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573068    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-etc-sysctl-d\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.573448 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573095    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e97b2da-120f-4a72-a6ea-907bcb72a13e-dbus\") pod \"global-pull-secret-syncer-jd8ww\" (UID: \"1e97b2da-120f-4a72-a6ea-907bcb72a13e\") " pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:31:59.573448 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573121    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.573448 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573140    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-etc-sysctl-conf\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.573448 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573153    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhs42\" (UniqueName: \"kubernetes.io/projected/11688621-db80-4166-80e2-b07c9755e04a-kube-api-access-lhs42\") pod \"iptables-alerter-2mmwh\" (UID: \"11688621-db80-4166-80e2-b07c9755e04a\") " pod="openshift-network-operator/iptables-alerter-2mmwh"
Sep 07 22:31:59.573448 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573175    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.573448 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573191    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5efbc01b-b5b9-40df-9ba9-71acdfd8be6e-konnectivity-ca\") pod \"konnectivity-agent-n6df9\" (UID: \"5efbc01b-b5b9-40df-9ba9-71acdfd8be6e\") " pod="kube-system/konnectivity-agent-n6df9"
Sep 07 22:31:59.573448 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573234    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-cnibin\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.573448 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573239    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-system-cni-dir\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.573448 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573195    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-cnibin\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.573448 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573195    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-run\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.573448 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573298    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-etc-systemd\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.574116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573464    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-var-lib-kubelet\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.574116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573499    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-host\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.574116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573523    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e97b2da-120f-4a72-a6ea-907bcb72a13e-kubelet-config\") pod \"global-pull-secret-syncer-jd8ww\" (UID: \"1e97b2da-120f-4a72-a6ea-907bcb72a13e\") " pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:31:59.574116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573533    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-etc-sysctl-d\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.574116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573573    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-lib-modules\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.574116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573610    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11688621-db80-4166-80e2-b07c9755e04a-iptables-alerter-script\") pod \"iptables-alerter-2mmwh\" (UID: \"11688621-db80-4166-80e2-b07c9755e04a\") " pod="openshift-network-operator/iptables-alerter-2mmwh"
Sep 07 22:31:59.574116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573617    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-sys\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.574116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573631    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-etc-sysconfig\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.574116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573651    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c64443d-b524-482c-99f9-3719f2a401c7-tmp\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.574116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573664    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e97b2da-120f-4a72-a6ea-907bcb72a13e-dbus\") pod \"global-pull-secret-syncer-jd8ww\" (UID: \"1e97b2da-120f-4a72-a6ea-907bcb72a13e\") " pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:31:59.574116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573672    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tznsr\" (UniqueName: \"kubernetes.io/projected/7c64443d-b524-482c-99f9-3719f2a401c7-kube-api-access-tznsr\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.574116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573688    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret\") pod \"global-pull-secret-syncer-jd8ww\" (UID: \"1e97b2da-120f-4a72-a6ea-907bcb72a13e\") " pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:31:59.574116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573752    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c64443d-b524-482c-99f9-3719f2a401c7-etc-tuned\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.574116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573779    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.574116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573816    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.574116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573840    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-etc-kubernetes\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.574116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.573878    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e97b2da-120f-4a72-a6ea-907bcb72a13e-kubelet-config\") pod \"global-pull-secret-syncer-jd8ww\" (UID: \"1e97b2da-120f-4a72-a6ea-907bcb72a13e\") " pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:31:59.574116 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:59.573948    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:31:59.574886 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:59.574007    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret podName:1e97b2da-120f-4a72-a6ea-907bcb72a13e nodeName:}" failed. No retries permitted until 2026-09-07 22:32:00.073989124 +0000 UTC m=+3.130822063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret") pod "global-pull-secret-syncer-jd8ww" (UID: "1e97b2da-120f-4a72-a6ea-907bcb72a13e") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:31:59.574886 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.574273    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.574886 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.574288    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-lib-modules\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.574886 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.574360    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-var-lib-kubelet\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.574886 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.574404    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-etc-sysconfig\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.574886 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.574415    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11688621-db80-4166-80e2-b07c9755e04a-iptables-alerter-script\") pod \"iptables-alerter-2mmwh\" (UID: \"11688621-db80-4166-80e2-b07c9755e04a\") " pod="openshift-network-operator/iptables-alerter-2mmwh"
Sep 07 22:31:59.574886 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.574448    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c64443d-b524-482c-99f9-3719f2a401c7-host\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.576542 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.576519    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5efbc01b-b5b9-40df-9ba9-71acdfd8be6e-agent-certs\") pod \"konnectivity-agent-n6df9\" (UID: \"5efbc01b-b5b9-40df-9ba9-71acdfd8be6e\") " pod="kube-system/konnectivity-agent-n6df9"
Sep 07 22:31:59.578266 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:59.578230    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:31:59.578383 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:59.578273    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:31:59.578383 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:59.578288    2567 projected.go:196] Error preparing data for projected volume kube-api-access-fz2t8 for pod openshift-network-diagnostics/network-check-target-q6zr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:31:59.578383 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:59.578346    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8 podName:0a7d1434-99e1-4ab9-8b17-08aebf32c93a nodeName:}" failed. No retries permitted until 2026-09-07 22:32:00.078328975 +0000 UTC m=+3.135161924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fz2t8" (UniqueName: "kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8") pod "network-check-target-q6zr5" (UID: "0a7d1434-99e1-4ab9-8b17-08aebf32c93a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:31:59.580660 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.580634    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7v6m\" (UniqueName: \"kubernetes.io/projected/d9a478e0-d256-4ab9-9e76-6a751a9b34f7-kube-api-access-c7v6m\") pod \"multus-additional-cni-plugins-x7bn4\" (UID: \"d9a478e0-d256-4ab9-9e76-6a751a9b34f7\") " pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.582305 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.582278    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tznsr\" (UniqueName: \"kubernetes.io/projected/7c64443d-b524-482c-99f9-3719f2a401c7-kube-api-access-tznsr\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.583039 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.583014    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhs42\" (UniqueName: \"kubernetes.io/projected/11688621-db80-4166-80e2-b07c9755e04a-kube-api-access-lhs42\") pod \"iptables-alerter-2mmwh\" (UID: \"11688621-db80-4166-80e2-b07c9755e04a\") " pod="openshift-network-operator/iptables-alerter-2mmwh"
Sep 07 22:31:59.583692 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.583672    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c64443d-b524-482c-99f9-3719f2a401c7-etc-tuned\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.585348 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.585317    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c64443d-b524-482c-99f9-3719f2a401c7-tmp\") pod \"tuned-kf2z2\" (UID: \"7c64443d-b524-482c-99f9-3719f2a401c7\") " pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.663511 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.663472    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vb75n"
Sep 07 22:31:59.673167 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.673142    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv"
Sep 07 22:31:59.685954 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.685928    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cz944"
Sep 07 22:31:59.691959 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.691935    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:31:59.700070 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.700047    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b8xx6"
Sep 07 22:31:59.708897 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.708867    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2mmwh"
Sep 07 22:31:59.716977 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.716938    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kf2z2"
Sep 07 22:31:59.724763 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.724741    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n6df9"
Sep 07 22:31:59.730521 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.730501    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x7bn4"
Sep 07 22:31:59.976862 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:31:59.976826    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs\") pod \"network-metrics-daemon-wrtvx\" (UID: \"f95816cb-c07f-4e9f-b828-930dcf1c06eb\") " pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:31:59.977060 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:59.976997    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:31:59.977125 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:31:59.977061    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs podName:f95816cb-c07f-4e9f-b828-930dcf1c06eb nodeName:}" failed. No retries permitted until 2026-09-07 22:32:00.977045579 +0000 UTC m=+4.033878523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs") pod "network-metrics-daemon-wrtvx" (UID: "f95816cb-c07f-4e9f-b828-930dcf1c06eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:00.077688 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:00.077638    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret\") pod \"global-pull-secret-syncer-jd8ww\" (UID: \"1e97b2da-120f-4a72-a6ea-907bcb72a13e\") " pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:00.077855 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:00.077814    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:00.077933 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:00.077897    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret podName:1e97b2da-120f-4a72-a6ea-907bcb72a13e nodeName:}" failed. No retries permitted until 2026-09-07 22:32:01.077877012 +0000 UTC m=+4.134709971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret") pod "global-pull-secret-syncer-jd8ww" (UID: "1e97b2da-120f-4a72-a6ea-907bcb72a13e") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:00.178870 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:00.178822    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz2t8\" (UniqueName: \"kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8\") pod \"network-check-target-q6zr5\" (UID: \"0a7d1434-99e1-4ab9-8b17-08aebf32c93a\") " pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:00.179037 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:00.178980    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:00.179037 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:00.179007    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:00.179037 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:00.179019    2567 projected.go:196] Error preparing data for projected volume kube-api-access-fz2t8 for pod openshift-network-diagnostics/network-check-target-q6zr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:00.179209 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:00.179086    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8 podName:0a7d1434-99e1-4ab9-8b17-08aebf32c93a nodeName:}" failed. No retries permitted until 2026-09-07 22:32:01.17906815 +0000 UTC m=+4.235901095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fz2t8" (UniqueName: "kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8") pod "network-check-target-q6zr5" (UID: "0a7d1434-99e1-4ab9-8b17-08aebf32c93a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:00.396338 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:00.396295    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 22:26:59 +0000 UTC" deadline="2028-03-09 04:23:59.534705336 +0000 UTC"
Sep 07 22:32:00.396338 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:00.396330    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13157h51m59.138378498s"
Sep 07 22:32:00.501715 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:32:00.501690    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c64443d_b524_482c_99f9_3719f2a401c7.slice/crio-562d0acca133e7d8a8c6a1ef73060d86883310f4780ec470546e653369a1cce2 WatchSource:0}: Error finding container 562d0acca133e7d8a8c6a1ef73060d86883310f4780ec470546e653369a1cce2: Status 404 returned error can't find the container with id 562d0acca133e7d8a8c6a1ef73060d86883310f4780ec470546e653369a1cce2
Sep 07 22:32:00.502826 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:32:00.502801    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafca2585_5d1b_4577_8efc_882374f40887.slice/crio-c8e3164e7f2e67a2c38ef02c87e927f34a4f331af9c127d455b952f1826c490e WatchSource:0}: Error finding container c8e3164e7f2e67a2c38ef02c87e927f34a4f331af9c127d455b952f1826c490e: Status 404 returned error can't find the container with id c8e3164e7f2e67a2c38ef02c87e927f34a4f331af9c127d455b952f1826c490e
Sep 07 22:32:00.505774 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:32:00.504177    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11688621_db80_4166_80e2_b07c9755e04a.slice/crio-5c5c3fad6d798528cde51f48cbcd5a15802c35401d612eb5136acdc0c31b90c0 WatchSource:0}: Error finding container 5c5c3fad6d798528cde51f48cbcd5a15802c35401d612eb5136acdc0c31b90c0: Status 404 returned error can't find the container with id 5c5c3fad6d798528cde51f48cbcd5a15802c35401d612eb5136acdc0c31b90c0
Sep 07 22:32:00.507464 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:32:00.507022    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a478e0_d256_4ab9_9e76_6a751a9b34f7.slice/crio-39235c7470869e4c9f3bd2ec8fac95747398c09cbb66f94378c4f4495de88e58 WatchSource:0}: Error finding container 39235c7470869e4c9f3bd2ec8fac95747398c09cbb66f94378c4f4495de88e58: Status 404 returned error can't find the container with id 39235c7470869e4c9f3bd2ec8fac95747398c09cbb66f94378c4f4495de88e58
Sep 07 22:32:00.509417 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:32:00.509336    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a27e39_3141_4765_8e18_d8195a6c567e.slice/crio-aa02072b48f172932e17c25e7ce8a0af3e64fd67ea55b8c1bda0fbe58fe3f35e WatchSource:0}: Error finding container aa02072b48f172932e17c25e7ce8a0af3e64fd67ea55b8c1bda0fbe58fe3f35e: Status 404 returned error can't find the container with id aa02072b48f172932e17c25e7ce8a0af3e64fd67ea55b8c1bda0fbe58fe3f35e
Sep 07 22:32:00.510019 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:32:00.509992    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2dfdc2_be64_43c6_813f_f34f09577b9b.slice/crio-a0e1cf0d87d386342c7729e69a482a63af66a02454fa95254b88c1494ff9fdbb WatchSource:0}: Error finding container a0e1cf0d87d386342c7729e69a482a63af66a02454fa95254b88c1494ff9fdbb: Status 404 returned error can't find the container with id a0e1cf0d87d386342c7729e69a482a63af66a02454fa95254b88c1494ff9fdbb
Sep 07 22:32:00.511156 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:32:00.511122    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09b22bc_14fb_423b_b7b4_0d3620da9bca.slice/crio-415312976d1a4d4ce13bd5ba2f1fddbfa65d038f664746aadd155c073b8e61ac WatchSource:0}: Error finding container 415312976d1a4d4ce13bd5ba2f1fddbfa65d038f664746aadd155c073b8e61ac: Status 404 returned error can't find the container with id 415312976d1a4d4ce13bd5ba2f1fddbfa65d038f664746aadd155c073b8e61ac
Sep 07 22:32:00.513714 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:32:00.512766    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d878f7_d905_4ad5_a824_bdf4a0f25476.slice/crio-92de8aa506bafd5984721df3dbd062e7fad93c0e1982b1a76f3058f672b4e8eb WatchSource:0}: Error finding container 92de8aa506bafd5984721df3dbd062e7fad93c0e1982b1a76f3058f672b4e8eb: Status 404 returned error can't find the container with id 92de8aa506bafd5984721df3dbd062e7fad93c0e1982b1a76f3058f672b4e8eb
Sep 07 22:32:00.985321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:00.984971    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs\") pod \"network-metrics-daemon-wrtvx\" (UID: \"f95816cb-c07f-4e9f-b828-930dcf1c06eb\") " pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:00.985321 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:00.985236    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:00.985321 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:00.985320    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs podName:f95816cb-c07f-4e9f-b828-930dcf1c06eb nodeName:}" failed. No retries permitted until 2026-09-07 22:32:02.985300228 +0000 UTC m=+6.042133189 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs") pod "network-metrics-daemon-wrtvx" (UID: "f95816cb-c07f-4e9f-b828-930dcf1c06eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:01.086236 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.086189    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret\") pod \"global-pull-secret-syncer-jd8ww\" (UID: \"1e97b2da-120f-4a72-a6ea-907bcb72a13e\") " pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:01.086446 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:01.086360    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:01.086446 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:01.086426    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret podName:1e97b2da-120f-4a72-a6ea-907bcb72a13e nodeName:}" failed. No retries permitted until 2026-09-07 22:32:03.086407232 +0000 UTC m=+6.143240177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret") pod "global-pull-secret-syncer-jd8ww" (UID: "1e97b2da-120f-4a72-a6ea-907bcb72a13e") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:01.187125 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.187083    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz2t8\" (UniqueName: \"kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8\") pod \"network-check-target-q6zr5\" (UID: \"0a7d1434-99e1-4ab9-8b17-08aebf32c93a\") " pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:01.187355 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:01.187307    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:01.187355 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:01.187331    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:01.187355 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:01.187345    2567 projected.go:196] Error preparing data for projected volume kube-api-access-fz2t8 for pod openshift-network-diagnostics/network-check-target-q6zr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:01.187570 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:01.187413    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8 podName:0a7d1434-99e1-4ab9-8b17-08aebf32c93a nodeName:}" failed. No retries permitted until 2026-09-07 22:32:03.187393714 +0000 UTC m=+6.244226659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fz2t8" (UniqueName: "kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8") pod "network-check-target-q6zr5" (UID: "0a7d1434-99e1-4ab9-8b17-08aebf32c93a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:01.397428 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.397271    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 22:26:59 +0000 UTC" deadline="2028-02-25 19:00:23.153491099 +0000 UTC"
Sep 07 22:32:01.397428 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.397336    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12860h28m21.756159306s"
Sep 07 22:32:01.461628 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.461595    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:01.461811 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:01.461724    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6zr5" podUID="0a7d1434-99e1-4ab9-8b17-08aebf32c93a"
Sep 07 22:32:01.462157 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.462137    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:01.462393 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.462372    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:01.462447 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:01.462427    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jd8ww" podUID="1e97b2da-120f-4a72-a6ea-907bcb72a13e"
Sep 07 22:32:01.462520 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:01.462490    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wrtvx" podUID="f95816cb-c07f-4e9f-b828-930dcf1c06eb"
Sep 07 22:32:01.485437 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.485396    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7" event={"ID":"0f2dfdc2-be64-43c6-813f-f34f09577b9b","Type":"ContainerStarted","Data":"a0e1cf0d87d386342c7729e69a482a63af66a02454fa95254b88c1494ff9fdbb"}
Sep 07 22:32:01.491722 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.491631    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b8xx6" event={"ID":"afca2585-5d1b-4577-8efc-882374f40887","Type":"ContainerStarted","Data":"c8e3164e7f2e67a2c38ef02c87e927f34a4f331af9c127d455b952f1826c490e"}
Sep 07 22:32:01.502177 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.501491    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal" event={"ID":"9c67f07165e3fd7605dfe27e9a9c951a","Type":"ContainerStarted","Data":"3c4dcc65b3af3babb3fb5a22130c3dc0444e9cd57b4290cc0fe51fd44761b70f"}
Sep 07 22:32:01.522007 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.521798    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n6df9" event={"ID":"5efbc01b-b5b9-40df-9ba9-71acdfd8be6e","Type":"ContainerStarted","Data":"fa710261b548968ad896ba9aba48f58b2343166ae8410529cb3e3974ebb15419"}
Sep 07 22:32:01.549427 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.549320    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cz944" event={"ID":"56a27e39-3141-4765-8e18-d8195a6c567e","Type":"ContainerStarted","Data":"aa02072b48f172932e17c25e7ce8a0af3e64fd67ea55b8c1bda0fbe58fe3f35e"}
Sep 07 22:32:01.564346 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.564262    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7bn4" event={"ID":"d9a478e0-d256-4ab9-9e76-6a751a9b34f7","Type":"ContainerStarted","Data":"39235c7470869e4c9f3bd2ec8fac95747398c09cbb66f94378c4f4495de88e58"}
Sep 07 22:32:01.569458 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.569422    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2mmwh" event={"ID":"11688621-db80-4166-80e2-b07c9755e04a","Type":"ContainerStarted","Data":"5c5c3fad6d798528cde51f48cbcd5a15802c35401d612eb5136acdc0c31b90c0"}
Sep 07 22:32:01.588542 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.588491    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kf2z2" event={"ID":"7c64443d-b524-482c-99f9-3719f2a401c7","Type":"ContainerStarted","Data":"562d0acca133e7d8a8c6a1ef73060d86883310f4780ec470546e653369a1cce2"}
Sep 07 22:32:01.605608 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.605570    2567 generic.go:356] "Generic (PLEG): container finished" podID="9c0cba75e87ce3cbaf5728c72e95a717" containerID="1ad17ae147ea2874dc44973735551461fc7fca03f8aed26a53d14a5aa8bf50b2" exitCode=0
Sep 07 22:32:01.605871 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.605846    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal" event={"ID":"9c0cba75e87ce3cbaf5728c72e95a717","Type":"ContainerDied","Data":"1ad17ae147ea2874dc44973735551461fc7fca03f8aed26a53d14a5aa8bf50b2"}
Sep 07 22:32:01.611864 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.611815    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv" event={"ID":"c9d878f7-d905-4ad5-a824-bdf4a0f25476","Type":"ContainerStarted","Data":"92de8aa506bafd5984721df3dbd062e7fad93c0e1982b1a76f3058f672b4e8eb"}
Sep 07 22:32:01.614922 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.614891    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vb75n" event={"ID":"b09b22bc-14fb-423b-b7b4-0d3620da9bca","Type":"ContainerStarted","Data":"415312976d1a4d4ce13bd5ba2f1fddbfa65d038f664746aadd155c073b8e61ac"}
Sep 07 22:32:01.620704 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:01.620650    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-99.ec2.internal" podStartSLOduration=3.6206293880000002 podStartE2EDuration="3.620629388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:01.521965047 +0000 UTC m=+4.578798009" watchObservedRunningTime="2026-09-07 22:32:01.620629388 +0000 UTC m=+4.677462350"
Sep 07 22:32:02.638277 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:02.638222    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal" event={"ID":"9c0cba75e87ce3cbaf5728c72e95a717","Type":"ContainerStarted","Data":"624712080c928904a6ad12b2ff0fc2d1f6d03e4f46c5de6d13abdadae0c7a338"}
Sep 07 22:32:02.653152 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:02.653091    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-99.ec2.internal" podStartSLOduration=4.653065796 podStartE2EDuration="4.653065796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:02.652687295 +0000 UTC m=+5.709520257" watchObservedRunningTime="2026-09-07 22:32:02.653065796 +0000 UTC m=+5.709898758"
Sep 07 22:32:03.004207 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:03.004119    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs\") pod \"network-metrics-daemon-wrtvx\" (UID: \"f95816cb-c07f-4e9f-b828-930dcf1c06eb\") " pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:03.004384 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:03.004287    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:03.004384 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:03.004353    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs podName:f95816cb-c07f-4e9f-b828-930dcf1c06eb nodeName:}" failed. No retries permitted until 2026-09-07 22:32:07.004334234 +0000 UTC m=+10.061167196 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs") pod "network-metrics-daemon-wrtvx" (UID: "f95816cb-c07f-4e9f-b828-930dcf1c06eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:03.105551 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:03.104903    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret\") pod \"global-pull-secret-syncer-jd8ww\" (UID: \"1e97b2da-120f-4a72-a6ea-907bcb72a13e\") " pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:03.105551 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:03.105128    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:03.105551 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:03.105191    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret podName:1e97b2da-120f-4a72-a6ea-907bcb72a13e nodeName:}" failed. No retries permitted until 2026-09-07 22:32:07.105174392 +0000 UTC m=+10.162007355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret") pod "global-pull-secret-syncer-jd8ww" (UID: "1e97b2da-120f-4a72-a6ea-907bcb72a13e") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:03.205850 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:03.205718    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz2t8\" (UniqueName: \"kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8\") pod \"network-check-target-q6zr5\" (UID: \"0a7d1434-99e1-4ab9-8b17-08aebf32c93a\") " pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:03.206024 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:03.205908    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:03.206024 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:03.205940    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:03.206024 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:03.205954    2567 projected.go:196] Error preparing data for projected volume kube-api-access-fz2t8 for pod openshift-network-diagnostics/network-check-target-q6zr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:03.206024 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:03.206022    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8 podName:0a7d1434-99e1-4ab9-8b17-08aebf32c93a nodeName:}" failed. No retries permitted until 2026-09-07 22:32:07.20600204 +0000 UTC m=+10.262834983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fz2t8" (UniqueName: "kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8") pod "network-check-target-q6zr5" (UID: "0a7d1434-99e1-4ab9-8b17-08aebf32c93a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:03.460308 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:03.460191    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:03.460308 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:03.460211    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:03.460308 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:03.460212    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:03.460590 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:03.460356    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jd8ww" podUID="1e97b2da-120f-4a72-a6ea-907bcb72a13e"
Sep 07 22:32:03.461165 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:03.460735    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wrtvx" podUID="f95816cb-c07f-4e9f-b828-930dcf1c06eb"
Sep 07 22:32:03.461165 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:03.460821    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6zr5" podUID="0a7d1434-99e1-4ab9-8b17-08aebf32c93a"
Sep 07 22:32:05.460401 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:05.460369    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:05.460855 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:05.460514    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jd8ww" podUID="1e97b2da-120f-4a72-a6ea-907bcb72a13e"
Sep 07 22:32:05.461200 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:05.460929    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:05.461200 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:05.461034    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wrtvx" podUID="f95816cb-c07f-4e9f-b828-930dcf1c06eb"
Sep 07 22:32:05.461200 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:05.461095    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:05.461200 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:05.461159    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6zr5" podUID="0a7d1434-99e1-4ab9-8b17-08aebf32c93a"
Sep 07 22:32:07.038588 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:07.038508    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs\") pod \"network-metrics-daemon-wrtvx\" (UID: \"f95816cb-c07f-4e9f-b828-930dcf1c06eb\") " pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:07.039175 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:07.038647    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:07.039175 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:07.038724    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs podName:f95816cb-c07f-4e9f-b828-930dcf1c06eb nodeName:}" failed. No retries permitted until 2026-09-07 22:32:15.038704502 +0000 UTC m=+18.095537457 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs") pod "network-metrics-daemon-wrtvx" (UID: "f95816cb-c07f-4e9f-b828-930dcf1c06eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:07.139716 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:07.139537    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret\") pod \"global-pull-secret-syncer-jd8ww\" (UID: \"1e97b2da-120f-4a72-a6ea-907bcb72a13e\") " pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:07.139716 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:07.139716    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:07.139963 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:07.139783    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret podName:1e97b2da-120f-4a72-a6ea-907bcb72a13e nodeName:}" failed. No retries permitted until 2026-09-07 22:32:15.139763532 +0000 UTC m=+18.196596495 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret") pod "global-pull-secret-syncer-jd8ww" (UID: "1e97b2da-120f-4a72-a6ea-907bcb72a13e") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:07.240311 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:07.240275    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz2t8\" (UniqueName: \"kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8\") pod \"network-check-target-q6zr5\" (UID: \"0a7d1434-99e1-4ab9-8b17-08aebf32c93a\") " pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:07.240507 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:07.240478    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:07.240507 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:07.240511    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:07.240662 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:07.240524    2567 projected.go:196] Error preparing data for projected volume kube-api-access-fz2t8 for pod openshift-network-diagnostics/network-check-target-q6zr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:07.240662 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:07.240581    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8 podName:0a7d1434-99e1-4ab9-8b17-08aebf32c93a nodeName:}" failed. No retries permitted until 2026-09-07 22:32:15.240564896 +0000 UTC m=+18.297397834 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fz2t8" (UniqueName: "kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8") pod "network-check-target-q6zr5" (UID: "0a7d1434-99e1-4ab9-8b17-08aebf32c93a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:07.461362 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:07.460971    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:07.461362 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:07.461074    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:07.461362 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:07.461116    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:07.461362 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:07.461121    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jd8ww" podUID="1e97b2da-120f-4a72-a6ea-907bcb72a13e"
Sep 07 22:32:07.461362 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:07.461209    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wrtvx" podUID="f95816cb-c07f-4e9f-b828-930dcf1c06eb"
Sep 07 22:32:07.461362 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:07.461326    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6zr5" podUID="0a7d1434-99e1-4ab9-8b17-08aebf32c93a"
Sep 07 22:32:09.459681 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:09.459493    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:09.459681 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:09.459502    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:09.459681 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:09.459624    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jd8ww" podUID="1e97b2da-120f-4a72-a6ea-907bcb72a13e"
Sep 07 22:32:09.459681 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:09.459634    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:09.460318 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:09.459975    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6zr5" podUID="0a7d1434-99e1-4ab9-8b17-08aebf32c93a"
Sep 07 22:32:09.460318 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:09.460081    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wrtvx" podUID="f95816cb-c07f-4e9f-b828-930dcf1c06eb"
Sep 07 22:32:11.459806 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:11.459439    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:11.459806 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:11.459480    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:11.459806 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:11.459602    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wrtvx" podUID="f95816cb-c07f-4e9f-b828-930dcf1c06eb"
Sep 07 22:32:11.459806 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:11.459674    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:11.459806 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:11.459668    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jd8ww" podUID="1e97b2da-120f-4a72-a6ea-907bcb72a13e"
Sep 07 22:32:11.459806 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:11.459754    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6zr5" podUID="0a7d1434-99e1-4ab9-8b17-08aebf32c93a"
Sep 07 22:32:13.459161 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:13.459125    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:13.459619 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:13.459120    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:13.459619 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:13.459268    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:13.459619 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:13.459292    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6zr5" podUID="0a7d1434-99e1-4ab9-8b17-08aebf32c93a"
Sep 07 22:32:13.459619 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:13.459379    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wrtvx" podUID="f95816cb-c07f-4e9f-b828-930dcf1c06eb"
Sep 07 22:32:13.459619 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:13.459476    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jd8ww" podUID="1e97b2da-120f-4a72-a6ea-907bcb72a13e"
Sep 07 22:32:15.107141 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:15.107105    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs\") pod \"network-metrics-daemon-wrtvx\" (UID: \"f95816cb-c07f-4e9f-b828-930dcf1c06eb\") " pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:15.107652 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:15.107280    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:15.107652 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:15.107351    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs podName:f95816cb-c07f-4e9f-b828-930dcf1c06eb nodeName:}" failed. No retries permitted until 2026-09-07 22:32:31.107335182 +0000 UTC m=+34.164168121 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs") pod "network-metrics-daemon-wrtvx" (UID: "f95816cb-c07f-4e9f-b828-930dcf1c06eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:15.207969 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:15.207933    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret\") pod \"global-pull-secret-syncer-jd8ww\" (UID: \"1e97b2da-120f-4a72-a6ea-907bcb72a13e\") " pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:15.208141 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:15.208074    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:15.208141 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:15.208135    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret podName:1e97b2da-120f-4a72-a6ea-907bcb72a13e nodeName:}" failed. No retries permitted until 2026-09-07 22:32:31.20811787 +0000 UTC m=+34.264950809 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret") pod "global-pull-secret-syncer-jd8ww" (UID: "1e97b2da-120f-4a72-a6ea-907bcb72a13e") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:15.308841 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:15.308800    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz2t8\" (UniqueName: \"kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8\") pod \"network-check-target-q6zr5\" (UID: \"0a7d1434-99e1-4ab9-8b17-08aebf32c93a\") " pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:15.309004 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:15.308944    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:15.309004 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:15.308964    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:15.309004 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:15.308978    2567 projected.go:196] Error preparing data for projected volume kube-api-access-fz2t8 for pod openshift-network-diagnostics/network-check-target-q6zr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:15.309153 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:15.309028    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8 podName:0a7d1434-99e1-4ab9-8b17-08aebf32c93a nodeName:}" failed. No retries permitted until 2026-09-07 22:32:31.309015003 +0000 UTC m=+34.365847942 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fz2t8" (UniqueName: "kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8") pod "network-check-target-q6zr5" (UID: "0a7d1434-99e1-4ab9-8b17-08aebf32c93a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:15.459354 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:15.459272    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:15.459512 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:15.459273    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:15.459512 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:15.459403    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wrtvx" podUID="f95816cb-c07f-4e9f-b828-930dcf1c06eb"
Sep 07 22:32:15.459512 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:15.459460    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6zr5" podUID="0a7d1434-99e1-4ab9-8b17-08aebf32c93a"
Sep 07 22:32:15.459512 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:15.459273    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:15.459695 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:15.459574    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jd8ww" podUID="1e97b2da-120f-4a72-a6ea-907bcb72a13e"
Sep 07 22:32:17.460769 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:17.460726    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:17.461266 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:17.460824    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:17.461266 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:17.460870    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:17.461266 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:17.460869    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jd8ww" podUID="1e97b2da-120f-4a72-a6ea-907bcb72a13e"
Sep 07 22:32:17.461266 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:17.460944    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6zr5" podUID="0a7d1434-99e1-4ab9-8b17-08aebf32c93a"
Sep 07 22:32:17.461266 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:17.461033    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wrtvx" podUID="f95816cb-c07f-4e9f-b828-930dcf1c06eb"
Sep 07 22:32:18.675518 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:18.675171    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7bn4" event={"ID":"d9a478e0-d256-4ab9-9e76-6a751a9b34f7","Type":"ContainerStarted","Data":"34ee52eb368d439f94ebe29256cfa4e678a47209fc8b5a3de8680bc3305ea6ce"}
Sep 07 22:32:18.677413 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:18.677345    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kf2z2" event={"ID":"7c64443d-b524-482c-99f9-3719f2a401c7","Type":"ContainerStarted","Data":"09ef622f4d9d86b0aa581657d1c5c06022a4677747bf6360bfbcb89427fb2600"}
Sep 07 22:32:18.679697 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:18.679673    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv" event={"ID":"c9d878f7-d905-4ad5-a824-bdf4a0f25476","Type":"ContainerStarted","Data":"467cbe0b81e2243e65217b6be45c7c9325a6479df1ab18c72fe7a188a6902b6d"}
Sep 07 22:32:18.681429 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:18.681381    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vb75n" event={"ID":"b09b22bc-14fb-423b-b7b4-0d3620da9bca","Type":"ContainerStarted","Data":"073b41ca224d5f3a118f8bdd404994ab118f7753f9f9e8e7831c8bf4eafec8bb"}
Sep 07 22:32:18.683613 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:18.683585    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7" event={"ID":"0f2dfdc2-be64-43c6-813f-f34f09577b9b","Type":"ContainerStarted","Data":"4898dd42f26b6dd97b0370a39fc66fefea37474879cf80d4453ad5e5a801afda"}
Sep 07 22:32:18.685539 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:18.685448    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b8xx6" event={"ID":"afca2585-5d1b-4577-8efc-882374f40887","Type":"ContainerStarted","Data":"fe93c30261318e96ac5d2ef8138c3ab3300bbc87a5ee45b9abf2db8154ed22c9"}
Sep 07 22:32:18.687117 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:18.687097    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n6df9" event={"ID":"5efbc01b-b5b9-40df-9ba9-71acdfd8be6e","Type":"ContainerStarted","Data":"4697b1c9478d40abbc753120ca37144de2084ed94cd9ab367544cddc70de99d3"}
Sep 07 22:32:18.689011 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:18.688985    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cz944" event={"ID":"56a27e39-3141-4765-8e18-d8195a6c567e","Type":"ContainerStarted","Data":"3eb92810f6df43512aaaf1eb6545aaf42b327c64bcfe7619133c8d080b2023a4"}
Sep 07 22:32:18.707096 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:18.707040    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-b8xx6" podStartSLOduration=3.724583276 podStartE2EDuration="21.707018316s" totalImagesPullingTime="17.98243504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:18.706479192 +0000 UTC m=+21.763312152" watchObservedRunningTime="2026-09-07 22:32:18.707018316 +0000 UTC m=+21.763851282"
Sep 07 22:32:18.719168 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:18.719126    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n6df9" podStartSLOduration=3.831182365 podStartE2EDuration="21.71910928s" totalImagesPullingTime="17.887926915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:18.718939423 +0000 UTC m=+21.775772385" watchObservedRunningTime="2026-09-07 22:32:18.71910928 +0000 UTC m=+21.775942241"
Sep 07 22:32:18.732870 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:18.732803    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vb75n" podStartSLOduration=3.846676598 podStartE2EDuration="21.732780401s" totalImagesPullingTime="17.886103803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:18.731895092 +0000 UTC m=+21.788728053" watchObservedRunningTime="2026-09-07 22:32:18.732780401 +0000 UTC m=+21.789613364"
Sep 07 22:32:18.752817 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:18.752748    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kf2z2" podStartSLOduration=3.856544839 podStartE2EDuration="21.752726379s" totalImagesPullingTime="17.89618154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:18.75205071 +0000 UTC m=+21.808883672" watchObservedRunningTime="2026-09-07 22:32:18.752726379 +0000 UTC m=+21.809559343"
Sep 07 22:32:18.765302 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:18.765234    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cz944" podStartSLOduration=3.87001958 podStartE2EDuration="21.765219124s" totalImagesPullingTime="17.895199544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:18.764958764 +0000 UTC m=+21.821791726" watchObservedRunningTime="2026-09-07 22:32:18.765219124 +0000 UTC m=+21.822052111"
Sep 07 22:32:19.459454 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:19.459423    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:19.459643 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:19.459555    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wrtvx" podUID="f95816cb-c07f-4e9f-b828-930dcf1c06eb"
Sep 07 22:32:19.459643 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:19.459620    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:19.459753 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:19.459686    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6zr5" podUID="0a7d1434-99e1-4ab9-8b17-08aebf32c93a"
Sep 07 22:32:19.460028 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:19.460000    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:19.460158 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:19.460124    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jd8ww" podUID="1e97b2da-120f-4a72-a6ea-907bcb72a13e"
Sep 07 22:32:19.692488 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:19.692395    2567 generic.go:356] "Generic (PLEG): container finished" podID="d9a478e0-d256-4ab9-9e76-6a751a9b34f7" containerID="34ee52eb368d439f94ebe29256cfa4e678a47209fc8b5a3de8680bc3305ea6ce" exitCode=0
Sep 07 22:32:19.693191 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:19.692487    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7bn4" event={"ID":"d9a478e0-d256-4ab9-9e76-6a751a9b34f7","Type":"ContainerDied","Data":"34ee52eb368d439f94ebe29256cfa4e678a47209fc8b5a3de8680bc3305ea6ce"}
Sep 07 22:32:19.696086 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:19.696056    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7" event={"ID":"0f2dfdc2-be64-43c6-813f-f34f09577b9b","Type":"ContainerStarted","Data":"d3dc413da29d58aa754d0147eb4cda61de83539cf7eb77aab3af6a8592694c69"}
Sep 07 22:32:19.696207 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:19.696136    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7" event={"ID":"0f2dfdc2-be64-43c6-813f-f34f09577b9b","Type":"ContainerStarted","Data":"1e2ea5e3997265fb14c57c53e8190c89979a4f1bf3c9dca3ba38388606ba0f4d"}
Sep 07 22:32:19.696207 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:19.696177    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7" event={"ID":"0f2dfdc2-be64-43c6-813f-f34f09577b9b","Type":"ContainerStarted","Data":"93240828a495f48ffd5e95d0db8a9c4eb1e5bf84a6857809e3acc23f83f3a4cf"}
Sep 07 22:32:19.696327 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:19.696210    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7" event={"ID":"0f2dfdc2-be64-43c6-813f-f34f09577b9b","Type":"ContainerStarted","Data":"d1989a9221807916e4757d5898f83a15474d4247fa2e3d9b3fb676bfdc31a152"}
Sep 07 22:32:19.696327 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:19.696263    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7" event={"ID":"0f2dfdc2-be64-43c6-813f-f34f09577b9b","Type":"ContainerStarted","Data":"e2388fa9f5f4a0335baa12df111628f77993f5e12bd9fb99c57496c3a73a1ffb"}
Sep 07 22:32:19.950811 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:19.950729    2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 07 22:32:20.430196 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:20.430090    2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-07T22:32:19.950747957Z","UUID":"9f33d058-7356-4a34-9c70-d655e793a167","Handler":null,"Name":"","Endpoint":""}
Sep 07 22:32:20.433035 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:20.433007    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 07 22:32:20.433178 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:20.433049    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 07 22:32:20.701484 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:20.701388    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv" event={"ID":"c9d878f7-d905-4ad5-a824-bdf4a0f25476","Type":"ContainerStarted","Data":"da6e85ea050b97893c3c9f26e3e1ee8868b708e06d4b9ed661e81e119b9c29fc"}
Sep 07 22:32:20.703308 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:20.703280    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2mmwh" event={"ID":"11688621-db80-4166-80e2-b07c9755e04a","Type":"ContainerStarted","Data":"e259281241da3508195d1ccd7a2658556846dbf403740951e7fce47ac60ea4fb"}
Sep 07 22:32:20.716151 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:20.716091    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2mmwh" podStartSLOduration=5.8143068289999995 podStartE2EDuration="23.7160687s" totalImagesPullingTime="17.901761871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:20.71572211 +0000 UTC m=+23.772555070" watchObservedRunningTime="2026-09-07 22:32:20.7160687 +0000 UTC m=+23.772901664"
Sep 07 22:32:21.459209 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:21.459026    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:21.459414 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:21.459026    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:21.459414 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:21.459319    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wrtvx" podUID="f95816cb-c07f-4e9f-b828-930dcf1c06eb"
Sep 07 22:32:21.459414 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:21.459026    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:21.459414 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:21.459366    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jd8ww" podUID="1e97b2da-120f-4a72-a6ea-907bcb72a13e"
Sep 07 22:32:21.459641 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:21.459420    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6zr5" podUID="0a7d1434-99e1-4ab9-8b17-08aebf32c93a"
Sep 07 22:32:21.637429 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:21.637383    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n6df9"
Sep 07 22:32:21.638068 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:21.638039    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n6df9"
Sep 07 22:32:21.708589 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:21.708552    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv" event={"ID":"c9d878f7-d905-4ad5-a824-bdf4a0f25476","Type":"ContainerStarted","Data":"ac4d218845161c4f4408f38ec1a6c9a0089aedc9d3883580d540e26db74e9013"}
Sep 07 22:32:21.712893 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:21.712815    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7" event={"ID":"0f2dfdc2-be64-43c6-813f-f34f09577b9b","Type":"ContainerStarted","Data":"ffc48d8d4562ac60925c0bd4f817cd14ac4ff9ae8abe0fc829e39665e36a7a1e"}
Sep 07 22:32:23.459280 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:23.459178    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:23.459715 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:23.459178    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:23.459715 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:23.459325    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6zr5" podUID="0a7d1434-99e1-4ab9-8b17-08aebf32c93a"
Sep 07 22:32:23.459715 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:23.459179    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:23.459715 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:23.459411    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jd8ww" podUID="1e97b2da-120f-4a72-a6ea-907bcb72a13e"
Sep 07 22:32:23.459715 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:23.459482    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wrtvx" podUID="f95816cb-c07f-4e9f-b828-930dcf1c06eb"
Sep 07 22:32:24.722087 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:24.721910    2567 generic.go:356] "Generic (PLEG): container finished" podID="d9a478e0-d256-4ab9-9e76-6a751a9b34f7" containerID="a0240f404794dd144ba007fe94e46cb09939f3f3ca37e4dbadaa74ce39512f0a" exitCode=0
Sep 07 22:32:24.722572 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:24.721996    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7bn4" event={"ID":"d9a478e0-d256-4ab9-9e76-6a751a9b34f7","Type":"ContainerDied","Data":"a0240f404794dd144ba007fe94e46cb09939f3f3ca37e4dbadaa74ce39512f0a"}
Sep 07 22:32:24.726256 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:24.726219    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7" event={"ID":"0f2dfdc2-be64-43c6-813f-f34f09577b9b","Type":"ContainerStarted","Data":"be320d76217a0c199ec831500b48f07cd2e522ef36d8b31074e356e396e17d4b"}
Sep 07 22:32:24.726473 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:24.726455    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:32:24.741370 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:24.741345    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:32:24.742308 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:24.742270    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5vqv" podStartSLOduration=7.70910327 podStartE2EDuration="27.742254343s" totalImagesPullingTime="20.033151073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:31:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:21.72460298 +0000 UTC m=+24.781435942" watchObservedRunningTime="2026-09-07 22:32:24.742254343 +0000 UTC m=+27.799087297"
Sep 07 22:32:24.766818 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:24.766763    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7" podStartSLOduration=9.824876542 podStartE2EDuration="27.766748403s" totalImagesPullingTime="17.941871861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:24.765739687 +0000 UTC m=+27.822572648" watchObservedRunningTime="2026-09-07 22:32:24.766748403 +0000 UTC m=+27.823581395"
Sep 07 22:32:25.459146 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:25.459112    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:25.459146 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:25.459133    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:25.459405 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:25.459123    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:25.459405 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:25.459219    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jd8ww" podUID="1e97b2da-120f-4a72-a6ea-907bcb72a13e"
Sep 07 22:32:25.459405 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:25.459300    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6zr5" podUID="0a7d1434-99e1-4ab9-8b17-08aebf32c93a"
Sep 07 22:32:25.459405 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:25.459369    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wrtvx" podUID="f95816cb-c07f-4e9f-b828-930dcf1c06eb"
Sep 07 22:32:25.728391 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:25.728323    2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 07 22:32:25.728902 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:25.728784    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:32:25.743864 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:25.743836    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:32:25.906132 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:25.906101    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jd8ww"]
Sep 07 22:32:25.906304 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:25.906218    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:25.906582 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:25.906337    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jd8ww" podUID="1e97b2da-120f-4a72-a6ea-907bcb72a13e"
Sep 07 22:32:25.915913 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:25.915655    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q6zr5"]
Sep 07 22:32:25.915913 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:25.915777    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:25.915913 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:25.915881    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6zr5" podUID="0a7d1434-99e1-4ab9-8b17-08aebf32c93a"
Sep 07 22:32:25.916371 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:25.916350    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wrtvx"]
Sep 07 22:32:25.916463 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:25.916448    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:25.916572 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:25.916553    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wrtvx" podUID="f95816cb-c07f-4e9f-b828-930dcf1c06eb"
Sep 07 22:32:26.732261 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:26.732208    2567 generic.go:356] "Generic (PLEG): container finished" podID="d9a478e0-d256-4ab9-9e76-6a751a9b34f7" containerID="55947506a97ebe29a776883c3de889215106c066fad791a3a55bb9096ac88659" exitCode=0
Sep 07 22:32:26.732795 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:26.732299    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7bn4" event={"ID":"d9a478e0-d256-4ab9-9e76-6a751a9b34f7","Type":"ContainerDied","Data":"55947506a97ebe29a776883c3de889215106c066fad791a3a55bb9096ac88659"}
Sep 07 22:32:26.732795 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:26.732375    2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 07 22:32:27.288329 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:27.288292    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:32:27.462135 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:27.462062    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:27.462294 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:27.462145    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6zr5" podUID="0a7d1434-99e1-4ab9-8b17-08aebf32c93a"
Sep 07 22:32:27.462294 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:27.462063    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:27.462294 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:27.462186    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:27.462440 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:27.462321    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wrtvx" podUID="f95816cb-c07f-4e9f-b828-930dcf1c06eb"
Sep 07 22:32:27.462440 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:27.462391    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jd8ww" podUID="1e97b2da-120f-4a72-a6ea-907bcb72a13e"
Sep 07 22:32:28.739955 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:28.739781    2567 generic.go:356] "Generic (PLEG): container finished" podID="d9a478e0-d256-4ab9-9e76-6a751a9b34f7" containerID="275f200f2ae1a06b3bd2fc538dd193971f945ae725caad40a31ce5105235d92a" exitCode=0
Sep 07 22:32:28.740455 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:28.739872    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7bn4" event={"ID":"d9a478e0-d256-4ab9-9e76-6a751a9b34f7","Type":"ContainerDied","Data":"275f200f2ae1a06b3bd2fc538dd193971f945ae725caad40a31ce5105235d92a"}
Sep 07 22:32:29.035923 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:29.035837    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n6df9"
Sep 07 22:32:29.036087 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:29.035991    2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 07 22:32:29.036522 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:29.036501    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n6df9"
Sep 07 22:32:29.459918 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:29.459884    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:29.460089 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:29.459884    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:29.460089 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:29.460018    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6zr5" podUID="0a7d1434-99e1-4ab9-8b17-08aebf32c93a"
Sep 07 22:32:29.460212 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:29.459884    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:29.460212 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:29.460152    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wrtvx" podUID="f95816cb-c07f-4e9f-b828-930dcf1c06eb"
Sep 07 22:32:29.460346 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:29.460321    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jd8ww" podUID="1e97b2da-120f-4a72-a6ea-907bcb72a13e"
Sep 07 22:32:31.130915 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.130877    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs\") pod \"network-metrics-daemon-wrtvx\" (UID: \"f95816cb-c07f-4e9f-b828-930dcf1c06eb\") " pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:31.131411 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:31.131061    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:31.131411 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:31.131145    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs podName:f95816cb-c07f-4e9f-b828-930dcf1c06eb nodeName:}" failed. No retries permitted until 2026-09-07 22:33:03.131124131 +0000 UTC m=+66.187957077 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs") pod "network-metrics-daemon-wrtvx" (UID: "f95816cb-c07f-4e9f-b828-930dcf1c06eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:31.231516 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.231425    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret\") pod \"global-pull-secret-syncer-jd8ww\" (UID: \"1e97b2da-120f-4a72-a6ea-907bcb72a13e\") " pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:31.231683 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:31.231532    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:31.231683 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:31.231591    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret podName:1e97b2da-120f-4a72-a6ea-907bcb72a13e nodeName:}" failed. No retries permitted until 2026-09-07 22:33:03.231571451 +0000 UTC m=+66.288404389 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret") pod "global-pull-secret-syncer-jd8ww" (UID: "1e97b2da-120f-4a72-a6ea-907bcb72a13e") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:31.235337 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.235312    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-99.ec2.internal" event="NodeReady"
Sep 07 22:32:31.235474 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.235460    2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 07 22:32:31.267870 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.267832    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79998dbc84-kskwc"]
Sep 07 22:32:31.285627 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.285592    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cwg2q"]
Sep 07 22:32:31.285797 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.285773    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.288100 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.288078    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 07 22:32:31.288238 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.288175    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 07 22:32:31.288497 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.288475    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hvdml\""
Sep 07 22:32:31.288610 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.288508    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 07 22:32:31.296851 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.296572    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 07 22:32:31.321263 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.321216    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79998dbc84-kskwc"]
Sep 07 22:32:31.321417 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.321279    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lxqwk"]
Sep 07 22:32:31.321479 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.321448    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:32:31.323658 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.323629    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 07 22:32:31.323791 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.323766    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zd2nk\""
Sep 07 22:32:31.323877 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.323862    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 07 22:32:31.332318 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.332287    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz2t8\" (UniqueName: \"kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8\") pod \"network-check-target-q6zr5\" (UID: \"0a7d1434-99e1-4ab9-8b17-08aebf32c93a\") " pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:31.332475 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:31.332458    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:31.332540 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:31.332479    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:31.332540 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:31.332489    2567 projected.go:196] Error preparing data for projected volume kube-api-access-fz2t8 for pod openshift-network-diagnostics/network-check-target-q6zr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:31.332637 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:31.332546    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8 podName:0a7d1434-99e1-4ab9-8b17-08aebf32c93a nodeName:}" failed. No retries permitted until 2026-09-07 22:33:03.332527571 +0000 UTC m=+66.389360521 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fz2t8" (UniqueName: "kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8") pod "network-check-target-q6zr5" (UID: "0a7d1434-99e1-4ab9-8b17-08aebf32c93a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:31.337276 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.337223    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cwg2q"]
Sep 07 22:32:31.337396 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.337301    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lxqwk"]
Sep 07 22:32:31.337396 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.337332    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lxqwk"
Sep 07 22:32:31.339874 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.339851    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 07 22:32:31.339987 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.339912    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 07 22:32:31.339987 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.339921    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zz6fl\""
Sep 07 22:32:31.339987 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.339960    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 07 22:32:31.433056 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.433006    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/90c9c42a-571f-4e9c-a741-c4ac016b1308-config-volume\") pod \"dns-default-cwg2q\" (UID: \"90c9c42a-571f-4e9c-a741-c4ac016b1308\") " pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:32:31.433056 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.433049    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhs5k\" (UniqueName: \"kubernetes.io/projected/90c9c42a-571f-4e9c-a741-c4ac016b1308-kube-api-access-hhs5k\") pod \"dns-default-cwg2q\" (UID: \"90c9c42a-571f-4e9c-a741-c4ac016b1308\") " pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:32:31.433290 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.433075    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.433290 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.433102    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6836bd00-5ba9-4802-b393-0d830fb1eba1-trusted-ca\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.433290 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.433120    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6836bd00-5ba9-4802-b393-0d830fb1eba1-installation-pull-secrets\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.433290 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.433149    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjftc\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-kube-api-access-fjftc\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.433290 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.433238    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8htl\" (UniqueName: \"kubernetes.io/projected/a7e100c0-fa8b-4565-978b-f3fa81227e5a-kube-api-access-g8htl\") pod \"ingress-canary-lxqwk\" (UID: \"a7e100c0-fa8b-4565-978b-f3fa81227e5a\") " pod="openshift-ingress-canary/ingress-canary-lxqwk"
Sep 07 22:32:31.433290 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.433285    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6836bd00-5ba9-4802-b393-0d830fb1eba1-image-registry-private-configuration\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.433526 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.433304    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6836bd00-5ba9-4802-b393-0d830fb1eba1-ca-trust-extracted\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.433526 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.433332    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls\") pod \"dns-default-cwg2q\" (UID: \"90c9c42a-571f-4e9c-a741-c4ac016b1308\") " pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:32:31.433526 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.433375    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7e100c0-fa8b-4565-978b-f3fa81227e5a-cert\") pod \"ingress-canary-lxqwk\" (UID: \"a7e100c0-fa8b-4565-978b-f3fa81227e5a\") " pod="openshift-ingress-canary/ingress-canary-lxqwk"
Sep 07 22:32:31.433526 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.433389    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-certificates\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.433526 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.433417    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90c9c42a-571f-4e9c-a741-c4ac016b1308-tmp-dir\") pod \"dns-default-cwg2q\" (UID: \"90c9c42a-571f-4e9c-a741-c4ac016b1308\") " pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:32:31.433526 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.433444    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-bound-sa-token\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.459469 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.459439    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:32:31.459607 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.459440    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:32:31.459662 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.459440    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:32:31.462473 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.462452    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 22:32:31.462473 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.462462    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lg5n9\""
Sep 07 22:32:31.462683 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.462454    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 22:32:31.462683 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.462452    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7qh6q\""
Sep 07 22:32:31.462683 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.462619    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 22:32:31.462849 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.462830    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 22:32:31.534211 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.534178    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8htl\" (UniqueName: \"kubernetes.io/projected/a7e100c0-fa8b-4565-978b-f3fa81227e5a-kube-api-access-g8htl\") pod \"ingress-canary-lxqwk\" (UID: \"a7e100c0-fa8b-4565-978b-f3fa81227e5a\") " pod="openshift-ingress-canary/ingress-canary-lxqwk"
Sep 07 22:32:31.534211 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.534213    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6836bd00-5ba9-4802-b393-0d830fb1eba1-image-registry-private-configuration\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.534462 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.534231    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6836bd00-5ba9-4802-b393-0d830fb1eba1-ca-trust-extracted\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.534462 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.534271    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls\") pod \"dns-default-cwg2q\" (UID: \"90c9c42a-571f-4e9c-a741-c4ac016b1308\") " pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:32:31.534462 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.534332    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7e100c0-fa8b-4565-978b-f3fa81227e5a-cert\") pod \"ingress-canary-lxqwk\" (UID: \"a7e100c0-fa8b-4565-978b-f3fa81227e5a\") " pod="openshift-ingress-canary/ingress-canary-lxqwk"
Sep 07 22:32:31.534462 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.534357    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-certificates\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.534462 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.534388    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90c9c42a-571f-4e9c-a741-c4ac016b1308-tmp-dir\") pod \"dns-default-cwg2q\" (UID: \"90c9c42a-571f-4e9c-a741-c4ac016b1308\") " pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:32:31.534462 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.534416    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-bound-sa-token\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.534462 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.534438    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/90c9c42a-571f-4e9c-a741-c4ac016b1308-config-volume\") pod \"dns-default-cwg2q\" (UID: \"90c9c42a-571f-4e9c-a741-c4ac016b1308\") " pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:32:31.534462 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.534454    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhs5k\" (UniqueName: \"kubernetes.io/projected/90c9c42a-571f-4e9c-a741-c4ac016b1308-kube-api-access-hhs5k\") pod \"dns-default-cwg2q\" (UID: \"90c9c42a-571f-4e9c-a741-c4ac016b1308\") " pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:32:31.534823 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.534470    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.534823 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.534494    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6836bd00-5ba9-4802-b393-0d830fb1eba1-trusted-ca\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.534823 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.534515    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6836bd00-5ba9-4802-b393-0d830fb1eba1-installation-pull-secrets\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.534823 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.534531    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjftc\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-kube-api-access-fjftc\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.534823 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.534703    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6836bd00-5ba9-4802-b393-0d830fb1eba1-ca-trust-extracted\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.535022 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:31.535000    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:32:31.535067 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:31.535060    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls podName:90c9c42a-571f-4e9c-a741-c4ac016b1308 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:32.035040566 +0000 UTC m=+35.091873521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls") pod "dns-default-cwg2q" (UID: "90c9c42a-571f-4e9c-a741-c4ac016b1308") : secret "dns-default-metrics-tls" not found
Sep 07 22:32:31.535345 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.535315    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/90c9c42a-571f-4e9c-a741-c4ac016b1308-config-volume\") pod \"dns-default-cwg2q\" (UID: \"90c9c42a-571f-4e9c-a741-c4ac016b1308\") " pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:32:31.535436 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:31.535406    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 22:32:31.535436 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:31.535419    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79998dbc84-kskwc: secret "image-registry-tls" not found
Sep 07 22:32:31.535524 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:31.535472    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls podName:6836bd00-5ba9-4802-b393-0d830fb1eba1 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:32.035453823 +0000 UTC m=+35.092286783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls") pod "image-registry-79998dbc84-kskwc" (UID: "6836bd00-5ba9-4802-b393-0d830fb1eba1") : secret "image-registry-tls" not found
Sep 07 22:32:31.535524 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.535487    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90c9c42a-571f-4e9c-a741-c4ac016b1308-tmp-dir\") pod \"dns-default-cwg2q\" (UID: \"90c9c42a-571f-4e9c-a741-c4ac016b1308\") " pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:32:31.535612 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.535530    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-certificates\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.536029 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.535953    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6836bd00-5ba9-4802-b393-0d830fb1eba1-trusted-ca\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.539456 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.539401    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6836bd00-5ba9-4802-b393-0d830fb1eba1-image-registry-private-configuration\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.539456 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.539421    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6836bd00-5ba9-4802-b393-0d830fb1eba1-installation-pull-secrets\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.539456 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.539428    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7e100c0-fa8b-4565-978b-f3fa81227e5a-cert\") pod \"ingress-canary-lxqwk\" (UID: \"a7e100c0-fa8b-4565-978b-f3fa81227e5a\") " pod="openshift-ingress-canary/ingress-canary-lxqwk"
Sep 07 22:32:31.543514 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.543488    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhs5k\" (UniqueName: \"kubernetes.io/projected/90c9c42a-571f-4e9c-a741-c4ac016b1308-kube-api-access-hhs5k\") pod \"dns-default-cwg2q\" (UID: \"90c9c42a-571f-4e9c-a741-c4ac016b1308\") " pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:32:31.543610 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.543552    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-bound-sa-token\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.544291 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.544269    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8htl\" (UniqueName: \"kubernetes.io/projected/a7e100c0-fa8b-4565-978b-f3fa81227e5a-kube-api-access-g8htl\") pod \"ingress-canary-lxqwk\" (UID: \"a7e100c0-fa8b-4565-978b-f3fa81227e5a\") " pod="openshift-ingress-canary/ingress-canary-lxqwk"
Sep 07 22:32:31.545042 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.545021    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjftc\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-kube-api-access-fjftc\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:31.647563 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:31.647488    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lxqwk"
Sep 07 22:32:32.038788 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:32.038750    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls\") pod \"dns-default-cwg2q\" (UID: \"90c9c42a-571f-4e9c-a741-c4ac016b1308\") " pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:32:32.038965 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:32.038917    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:32:32.039019 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:32.038922    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:32.039019 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:32.038995    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 22:32:32.039019 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:32.039012    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79998dbc84-kskwc: secret "image-registry-tls" not found
Sep 07 22:32:32.039134 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:32.038998    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls podName:90c9c42a-571f-4e9c-a741-c4ac016b1308 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:33.038974155 +0000 UTC m=+36.095807095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls") pod "dns-default-cwg2q" (UID: "90c9c42a-571f-4e9c-a741-c4ac016b1308") : secret "dns-default-metrics-tls" not found
Sep 07 22:32:32.039134 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:32.039076    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls podName:6836bd00-5ba9-4802-b393-0d830fb1eba1 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:33.039061439 +0000 UTC m=+36.095894377 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls") pod "image-registry-79998dbc84-kskwc" (UID: "6836bd00-5ba9-4802-b393-0d830fb1eba1") : secret "image-registry-tls" not found
Sep 07 22:32:33.048194 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:33.048027    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls\") pod \"dns-default-cwg2q\" (UID: \"90c9c42a-571f-4e9c-a741-c4ac016b1308\") " pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:32:33.048194 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:33.048183    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:32:33.048627 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:33.048237    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:33.048627 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:33.048297    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls podName:90c9c42a-571f-4e9c-a741-c4ac016b1308 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:35.048275199 +0000 UTC m=+38.105108154 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls") pod "dns-default-cwg2q" (UID: "90c9c42a-571f-4e9c-a741-c4ac016b1308") : secret "dns-default-metrics-tls" not found
Sep 07 22:32:33.048627 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:33.048346    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 22:32:33.048627 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:33.048363    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79998dbc84-kskwc: secret "image-registry-tls" not found
Sep 07 22:32:33.048627 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:33.048414    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls podName:6836bd00-5ba9-4802-b393-0d830fb1eba1 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:35.048395899 +0000 UTC m=+38.105228838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls") pod "image-registry-79998dbc84-kskwc" (UID: "6836bd00-5ba9-4802-b393-0d830fb1eba1") : secret "image-registry-tls" not found
Sep 07 22:32:33.309747 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:33.309674    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lxqwk"]
Sep 07 22:32:33.314774 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:32:33.314736    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e100c0_fa8b_4565_978b_f3fa81227e5a.slice/crio-199bdeb5cae452fad23db0d29d14667b1b5bbbe94f6604872528668855f0d5b2 WatchSource:0}: Error finding container 199bdeb5cae452fad23db0d29d14667b1b5bbbe94f6604872528668855f0d5b2: Status 404 returned error can't find the container with id 199bdeb5cae452fad23db0d29d14667b1b5bbbe94f6604872528668855f0d5b2
Sep 07 22:32:33.752999 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:33.752957    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lxqwk" event={"ID":"a7e100c0-fa8b-4565-978b-f3fa81227e5a","Type":"ContainerStarted","Data":"199bdeb5cae452fad23db0d29d14667b1b5bbbe94f6604872528668855f0d5b2"}
Sep 07 22:32:33.755738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:33.755707    2567 generic.go:356] "Generic (PLEG): container finished" podID="d9a478e0-d256-4ab9-9e76-6a751a9b34f7" containerID="5891f0b276d2e3446025452d0f21260e0de539a4c14fa6f7c83736aa7ab14e80" exitCode=0
Sep 07 22:32:33.755866 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:33.755755    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7bn4" event={"ID":"d9a478e0-d256-4ab9-9e76-6a751a9b34f7","Type":"ContainerDied","Data":"5891f0b276d2e3446025452d0f21260e0de539a4c14fa6f7c83736aa7ab14e80"}
Sep 07 22:32:34.761266 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:34.761221    2567 generic.go:356] "Generic (PLEG): container finished" podID="d9a478e0-d256-4ab9-9e76-6a751a9b34f7" containerID="ba85c5ba51e0bd21978922653ab8e71ebb58963921fb59851003e1a1f8ed6ad8" exitCode=0
Sep 07 22:32:34.761934 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:34.761273    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7bn4" event={"ID":"d9a478e0-d256-4ab9-9e76-6a751a9b34f7","Type":"ContainerDied","Data":"ba85c5ba51e0bd21978922653ab8e71ebb58963921fb59851003e1a1f8ed6ad8"}
Sep 07 22:32:35.064194 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:35.064158    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:35.064366 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:35.064232    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls\") pod \"dns-default-cwg2q\" (UID: \"90c9c42a-571f-4e9c-a741-c4ac016b1308\") " pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:32:35.064366 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:35.064318    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 22:32:35.064366 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:35.064337    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79998dbc84-kskwc: secret "image-registry-tls" not found
Sep 07 22:32:35.064366 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:35.064350    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:32:35.064596 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:35.064401    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls podName:6836bd00-5ba9-4802-b393-0d830fb1eba1 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:39.064384601 +0000 UTC m=+42.121217543 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls") pod "image-registry-79998dbc84-kskwc" (UID: "6836bd00-5ba9-4802-b393-0d830fb1eba1") : secret "image-registry-tls" not found
Sep 07 22:32:35.064596 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:35.064425    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls podName:90c9c42a-571f-4e9c-a741-c4ac016b1308 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:39.064417675 +0000 UTC m=+42.121250613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls") pod "dns-default-cwg2q" (UID: "90c9c42a-571f-4e9c-a741-c4ac016b1308") : secret "dns-default-metrics-tls" not found
Sep 07 22:32:35.767177 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:35.766958    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7bn4" event={"ID":"d9a478e0-d256-4ab9-9e76-6a751a9b34f7","Type":"ContainerStarted","Data":"64e012de68ac98ae1075902fcd51dcb929d5130322a9ae7711b0fbfe6230e3c5"}
Sep 07 22:32:35.789263 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:35.789198    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x7bn4" podStartSLOduration=9.666480064 podStartE2EDuration="38.789182369s" totalImagesPullingTime="29.122702305s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:31:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:35.789084224 +0000 UTC m=+38.845917187" watchObservedRunningTime="2026-09-07 22:32:35.789182369 +0000 UTC m=+38.846015329"
Sep 07 22:32:36.774236 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:36.774197    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lxqwk" event={"ID":"a7e100c0-fa8b-4565-978b-f3fa81227e5a","Type":"ContainerStarted","Data":"3920b93dfa8fd0883f6167394066454296f25a58aa043d395670dd3409e5ccfd"}
Sep 07 22:32:36.788394 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:36.788342    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lxqwk" podStartSLOduration=2.971460087 podStartE2EDuration="5.788327879s" totalImagesPullingTime="2.816867792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:36.787487737 +0000 UTC m=+39.844320699" watchObservedRunningTime="2026-09-07 22:32:36.788327879 +0000 UTC m=+39.845160840"
Sep 07 22:32:39.096336 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:39.096291    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls\") pod \"dns-default-cwg2q\" (UID: \"90c9c42a-571f-4e9c-a741-c4ac016b1308\") " pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:32:39.096727 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:39.096373    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:39.096727 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:39.096471    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:32:39.096727 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:39.096541    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls podName:90c9c42a-571f-4e9c-a741-c4ac016b1308 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:47.09652296 +0000 UTC m=+50.153355898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls") pod "dns-default-cwg2q" (UID: "90c9c42a-571f-4e9c-a741-c4ac016b1308") : secret "dns-default-metrics-tls" not found
Sep 07 22:32:39.096727 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:39.096479    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 22:32:39.096727 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:39.096575    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79998dbc84-kskwc: secret "image-registry-tls" not found
Sep 07 22:32:39.096727 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:39.096629    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls podName:6836bd00-5ba9-4802-b393-0d830fb1eba1 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:47.096615349 +0000 UTC m=+50.153448308 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls") pod "image-registry-79998dbc84-kskwc" (UID: "6836bd00-5ba9-4802-b393-0d830fb1eba1") : secret "image-registry-tls" not found
Sep 07 22:32:47.154859 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:47.154819    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls\") pod \"dns-default-cwg2q\" (UID: \"90c9c42a-571f-4e9c-a741-c4ac016b1308\") " pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:32:47.155349 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:47.154896    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:32:47.155349 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:47.154980    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 22:32:47.155349 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:47.154994    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79998dbc84-kskwc: secret "image-registry-tls" not found
Sep 07 22:32:47.155349 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:47.155049    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls podName:6836bd00-5ba9-4802-b393-0d830fb1eba1 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:03.155036772 +0000 UTC m=+66.211869711 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls") pod "image-registry-79998dbc84-kskwc" (UID: "6836bd00-5ba9-4802-b393-0d830fb1eba1") : secret "image-registry-tls" not found
Sep 07 22:32:47.155349 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:47.154979    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:32:47.155349 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:32:47.155121    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls podName:90c9c42a-571f-4e9c-a741-c4ac016b1308 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:03.15510664 +0000 UTC m=+66.211939579 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls") pod "dns-default-cwg2q" (UID: "90c9c42a-571f-4e9c-a741-c4ac016b1308") : secret "dns-default-metrics-tls" not found
Sep 07 22:32:58.750526 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:32:58.750491    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fnj7"
Sep 07 22:33:03.166000 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:03.165958    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:33:03.166526 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:03.166009    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs\") pod \"network-metrics-daemon-wrtvx\" (UID: \"f95816cb-c07f-4e9f-b828-930dcf1c06eb\") " pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:33:03.166526 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:03.166125    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 22:33:03.166526 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:03.166138    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls\") pod \"dns-default-cwg2q\" (UID: \"90c9c42a-571f-4e9c-a741-c4ac016b1308\") " pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:33:03.166526 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:03.166148    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79998dbc84-kskwc: secret "image-registry-tls" not found
Sep 07 22:33:03.166526 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:03.166222    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:33:03.166526 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:03.166273    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls podName:6836bd00-5ba9-4802-b393-0d830fb1eba1 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:35.166236342 +0000 UTC m=+98.223069281 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls") pod "image-registry-79998dbc84-kskwc" (UID: "6836bd00-5ba9-4802-b393-0d830fb1eba1") : secret "image-registry-tls" not found
Sep 07 22:33:03.166526 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:03.166290    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls podName:90c9c42a-571f-4e9c-a741-c4ac016b1308 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:35.166283335 +0000 UTC m=+98.223116273 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls") pod "dns-default-cwg2q" (UID: "90c9c42a-571f-4e9c-a741-c4ac016b1308") : secret "dns-default-metrics-tls" not found
Sep 07 22:33:03.168082 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:03.168062    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 22:33:03.177149 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:03.177125    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 07 22:33:03.177209 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:03.177195    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs podName:f95816cb-c07f-4e9f-b828-930dcf1c06eb nodeName:}" failed. No retries permitted until 2026-09-07 22:34:07.177175556 +0000 UTC m=+130.234008497 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs") pod "network-metrics-daemon-wrtvx" (UID: "f95816cb-c07f-4e9f-b828-930dcf1c06eb") : secret "metrics-daemon-secret" not found
Sep 07 22:33:03.267088 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:03.267053    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret\") pod \"global-pull-secret-syncer-jd8ww\" (UID: \"1e97b2da-120f-4a72-a6ea-907bcb72a13e\") " pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:33:03.269178 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:03.269152    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 22:33:03.281197 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:03.281169    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e97b2da-120f-4a72-a6ea-907bcb72a13e-original-pull-secret\") pod \"global-pull-secret-syncer-jd8ww\" (UID: \"1e97b2da-120f-4a72-a6ea-907bcb72a13e\") " pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:33:03.367921 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:03.367878    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz2t8\" (UniqueName: \"kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8\") pod \"network-check-target-q6zr5\" (UID: \"0a7d1434-99e1-4ab9-8b17-08aebf32c93a\") " pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:33:03.370050 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:03.370032    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 22:33:03.379912 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:03.379888    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 22:33:03.390777 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:03.390748    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz2t8\" (UniqueName: \"kubernetes.io/projected/0a7d1434-99e1-4ab9-8b17-08aebf32c93a-kube-api-access-fz2t8\") pod \"network-check-target-q6zr5\" (UID: \"0a7d1434-99e1-4ab9-8b17-08aebf32c93a\") " pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:33:03.574308 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:03.574276    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7qh6q\""
Sep 07 22:33:03.580149 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:03.580119    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jd8ww"
Sep 07 22:33:03.583167 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:03.583139    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:33:03.856824 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:03.856745    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jd8ww"]
Sep 07 22:33:03.859536 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:03.859509    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q6zr5"]
Sep 07 22:33:03.860467 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:33:03.860436    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e97b2da_120f_4a72_a6ea_907bcb72a13e.slice/crio-1ccd83c5450c4f99f1ad2029e6e38ec3e62c35545c91c883b7095563b2c37162 WatchSource:0}: Error finding container 1ccd83c5450c4f99f1ad2029e6e38ec3e62c35545c91c883b7095563b2c37162: Status 404 returned error can't find the container with id 1ccd83c5450c4f99f1ad2029e6e38ec3e62c35545c91c883b7095563b2c37162
Sep 07 22:33:03.862739 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:33:03.862716    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7d1434_99e1_4ab9_8b17_08aebf32c93a.slice/crio-6e449082884c07bd698b11c3b360170d822d90efb9948af854b4ce331a37334c WatchSource:0}: Error finding container 6e449082884c07bd698b11c3b360170d822d90efb9948af854b4ce331a37334c: Status 404 returned error can't find the container with id 6e449082884c07bd698b11c3b360170d822d90efb9948af854b4ce331a37334c
Sep 07 22:33:04.847238 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:04.847181    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q6zr5" event={"ID":"0a7d1434-99e1-4ab9-8b17-08aebf32c93a","Type":"ContainerStarted","Data":"6e449082884c07bd698b11c3b360170d822d90efb9948af854b4ce331a37334c"}
Sep 07 22:33:04.848545 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:04.848513    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jd8ww" event={"ID":"1e97b2da-120f-4a72-a6ea-907bcb72a13e","Type":"ContainerStarted","Data":"1ccd83c5450c4f99f1ad2029e6e38ec3e62c35545c91c883b7095563b2c37162"}
Sep 07 22:33:07.894609 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.894572    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-nrwrg"]
Sep 07 22:33:07.897360 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.897331    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:07.899764 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.899433    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 07 22:33:07.899764 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.899578    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xk276\""
Sep 07 22:33:07.899965 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.899817    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 07 22:33:07.900353 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.900330    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 07 22:33:07.900581 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.900561    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 07 22:33:07.909295 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.909263    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 07 22:33:07.912378 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.912349    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-nrwrg"]
Sep 07 22:33:07.986211 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.986173    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"]
Sep 07 22:33:07.989145 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.989114    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"
Sep 07 22:33:07.992155 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.992020    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 07 22:33:07.992155 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.992111    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 07 22:33:07.992155 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.992142    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cc2q7\""
Sep 07 22:33:07.992766 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.992287    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 07 22:33:07.992766 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.992309    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w"]
Sep 07 22:33:07.992766 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.992494    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 07 22:33:07.995535 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.995516    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w"
Sep 07 22:33:07.998212 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.997801    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 07 22:33:07.998212 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.997822    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5p6d2\""
Sep 07 22:33:07.998212 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.998156    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 07 22:33:07.998212 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:07.998184    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 07 22:33:08.000294 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.000272    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"]
Sep 07 22:33:08.006382 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.006356    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w"]
Sep 07 22:33:08.009780 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.009757    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77tck\" (UniqueName: \"kubernetes.io/projected/d3159cd3-b8ca-403e-a8c2-687db53940ab-kube-api-access-77tck\") pod \"console-operator-85b454874b-nrwrg\" (UID: \"d3159cd3-b8ca-403e-a8c2-687db53940ab\") " pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:08.009899 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.009812    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3159cd3-b8ca-403e-a8c2-687db53940ab-trusted-ca\") pod \"console-operator-85b454874b-nrwrg\" (UID: \"d3159cd3-b8ca-403e-a8c2-687db53940ab\") " pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:08.009899 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.009838    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d3159cd3-b8ca-403e-a8c2-687db53940ab-config\") pod \"console-operator-85b454874b-nrwrg\" (UID: \"d3159cd3-b8ca-403e-a8c2-687db53940ab\") " pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:08.009899 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.009863    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d3159cd3-b8ca-403e-a8c2-687db53940ab-serving-cert\") pod \"console-operator-85b454874b-nrwrg\" (UID: \"d3159cd3-b8ca-403e-a8c2-687db53940ab\") " pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:08.110334 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.110294    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfqnf\" (UniqueName: \"kubernetes.io/projected/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-kube-api-access-lfqnf\") pod \"cluster-samples-operator-7cdcfcf76c-nzk9w\" (UID: \"e3b9b281-ba77-4b3b-85ff-21e3632ff4ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w"
Sep 07 22:33:08.110334 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.110332    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-28hz4\" (UID: \"3cd41b43-6357-4ec5-b4b1-0cfc785ea58b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"
Sep 07 22:33:08.110551 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.110373    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77tck\" (UniqueName: \"kubernetes.io/projected/d3159cd3-b8ca-403e-a8c2-687db53940ab-kube-api-access-77tck\") pod \"console-operator-85b454874b-nrwrg\" (UID: \"d3159cd3-b8ca-403e-a8c2-687db53940ab\") " pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:08.110551 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.110403    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nzk9w\" (UID: \"e3b9b281-ba77-4b3b-85ff-21e3632ff4ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w"
Sep 07 22:33:08.110551 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.110456    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3159cd3-b8ca-403e-a8c2-687db53940ab-trusted-ca\") pod \"console-operator-85b454874b-nrwrg\" (UID: \"d3159cd3-b8ca-403e-a8c2-687db53940ab\") " pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:08.110551 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.110503    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d3159cd3-b8ca-403e-a8c2-687db53940ab-config\") pod \"console-operator-85b454874b-nrwrg\" (UID: \"d3159cd3-b8ca-403e-a8c2-687db53940ab\") " pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:08.110551 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.110538    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d3159cd3-b8ca-403e-a8c2-687db53940ab-serving-cert\") pod \"console-operator-85b454874b-nrwrg\" (UID: \"d3159cd3-b8ca-403e-a8c2-687db53940ab\") " pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:08.110744 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.110583    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-28hz4\" (UID: \"3cd41b43-6357-4ec5-b4b1-0cfc785ea58b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"
Sep 07 22:33:08.110744 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.110622    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4wbq\" (UniqueName: \"kubernetes.io/projected/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-kube-api-access-s4wbq\") pod \"cluster-monitoring-operator-6cdcc96965-28hz4\" (UID: \"3cd41b43-6357-4ec5-b4b1-0cfc785ea58b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"
Sep 07 22:33:08.111236 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.111208    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d3159cd3-b8ca-403e-a8c2-687db53940ab-config\") pod \"console-operator-85b454874b-nrwrg\" (UID: \"d3159cd3-b8ca-403e-a8c2-687db53940ab\") " pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:08.111409 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.111385    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3159cd3-b8ca-403e-a8c2-687db53940ab-trusted-ca\") pod \"console-operator-85b454874b-nrwrg\" (UID: \"d3159cd3-b8ca-403e-a8c2-687db53940ab\") " pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:08.113156 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.113133    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d3159cd3-b8ca-403e-a8c2-687db53940ab-serving-cert\") pod \"console-operator-85b454874b-nrwrg\" (UID: \"d3159cd3-b8ca-403e-a8c2-687db53940ab\") " pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:08.122338 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.122305    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77tck\" (UniqueName: \"kubernetes.io/projected/d3159cd3-b8ca-403e-a8c2-687db53940ab-kube-api-access-77tck\") pod \"console-operator-85b454874b-nrwrg\" (UID: \"d3159cd3-b8ca-403e-a8c2-687db53940ab\") " pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:08.209185 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.209096    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:08.211441 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.211407    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfqnf\" (UniqueName: \"kubernetes.io/projected/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-kube-api-access-lfqnf\") pod \"cluster-samples-operator-7cdcfcf76c-nzk9w\" (UID: \"e3b9b281-ba77-4b3b-85ff-21e3632ff4ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w"
Sep 07 22:33:08.211575 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.211455    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-28hz4\" (UID: \"3cd41b43-6357-4ec5-b4b1-0cfc785ea58b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"
Sep 07 22:33:08.211575 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.211495    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nzk9w\" (UID: \"e3b9b281-ba77-4b3b-85ff-21e3632ff4ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w"
Sep 07 22:33:08.211687 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.211572    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-28hz4\" (UID: \"3cd41b43-6357-4ec5-b4b1-0cfc785ea58b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"
Sep 07 22:33:08.211687 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.211600    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4wbq\" (UniqueName: \"kubernetes.io/projected/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-kube-api-access-s4wbq\") pod \"cluster-monitoring-operator-6cdcc96965-28hz4\" (UID: \"3cd41b43-6357-4ec5-b4b1-0cfc785ea58b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"
Sep 07 22:33:08.211794 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:08.211710    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 07 22:33:08.211794 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:08.211710    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 07 22:33:08.211794 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:08.211794    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-samples-operator-tls podName:e3b9b281-ba77-4b3b-85ff-21e3632ff4ce nodeName:}" failed. No retries permitted until 2026-09-07 22:33:08.711770501 +0000 UTC m=+71.768603463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nzk9w" (UID: "e3b9b281-ba77-4b3b-85ff-21e3632ff4ce") : secret "samples-operator-tls" not found
Sep 07 22:33:08.211950 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:08.211835    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls podName:3cd41b43-6357-4ec5-b4b1-0cfc785ea58b nodeName:}" failed. No retries permitted until 2026-09-07 22:33:08.71182479 +0000 UTC m=+71.768657734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-28hz4" (UID: "3cd41b43-6357-4ec5-b4b1-0cfc785ea58b") : secret "cluster-monitoring-operator-tls" not found
Sep 07 22:33:08.212583 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.212351    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-28hz4\" (UID: \"3cd41b43-6357-4ec5-b4b1-0cfc785ea58b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"
Sep 07 22:33:08.220504 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.220472    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4wbq\" (UniqueName: \"kubernetes.io/projected/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-kube-api-access-s4wbq\") pod \"cluster-monitoring-operator-6cdcc96965-28hz4\" (UID: \"3cd41b43-6357-4ec5-b4b1-0cfc785ea58b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"
Sep 07 22:33:08.220642 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.220567    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfqnf\" (UniqueName: \"kubernetes.io/projected/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-kube-api-access-lfqnf\") pod \"cluster-samples-operator-7cdcfcf76c-nzk9w\" (UID: \"e3b9b281-ba77-4b3b-85ff-21e3632ff4ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w"
Sep 07 22:33:08.715479 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.715436    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-28hz4\" (UID: \"3cd41b43-6357-4ec5-b4b1-0cfc785ea58b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"
Sep 07 22:33:08.715742 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:08.715520    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nzk9w\" (UID: \"e3b9b281-ba77-4b3b-85ff-21e3632ff4ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w"
Sep 07 22:33:08.715742 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:08.715608    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 07 22:33:08.715742 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:08.715621    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 07 22:33:08.715742 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:08.715701    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls podName:3cd41b43-6357-4ec5-b4b1-0cfc785ea58b nodeName:}" failed. No retries permitted until 2026-09-07 22:33:09.715684134 +0000 UTC m=+72.772517099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-28hz4" (UID: "3cd41b43-6357-4ec5-b4b1-0cfc785ea58b") : secret "cluster-monitoring-operator-tls" not found
Sep 07 22:33:08.715742 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:08.715741    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-samples-operator-tls podName:e3b9b281-ba77-4b3b-85ff-21e3632ff4ce nodeName:}" failed. No retries permitted until 2026-09-07 22:33:09.715728431 +0000 UTC m=+72.772561371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nzk9w" (UID: "e3b9b281-ba77-4b3b-85ff-21e3632ff4ce") : secret "samples-operator-tls" not found
Sep 07 22:33:09.153651 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:09.153609    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-nrwrg"]
Sep 07 22:33:09.157048 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:33:09.157019    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3159cd3_b8ca_403e_a8c2_687db53940ab.slice/crio-62535a314c8802b1322280f7893c95a390b676394ae0b2db44430bcb1d02aeb1 WatchSource:0}: Error finding container 62535a314c8802b1322280f7893c95a390b676394ae0b2db44430bcb1d02aeb1: Status 404 returned error can't find the container with id 62535a314c8802b1322280f7893c95a390b676394ae0b2db44430bcb1d02aeb1
Sep 07 22:33:09.724118 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:09.724071    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nzk9w\" (UID: \"e3b9b281-ba77-4b3b-85ff-21e3632ff4ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w"
Sep 07 22:33:09.724342 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:09.724227    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 07 22:33:09.724342 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:09.724259    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-28hz4\" (UID: \"3cd41b43-6357-4ec5-b4b1-0cfc785ea58b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"
Sep 07 22:33:09.724342 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:09.724304    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-samples-operator-tls podName:e3b9b281-ba77-4b3b-85ff-21e3632ff4ce nodeName:}" failed. No retries permitted until 2026-09-07 22:33:11.724289005 +0000 UTC m=+74.781121947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nzk9w" (UID: "e3b9b281-ba77-4b3b-85ff-21e3632ff4ce") : secret "samples-operator-tls" not found
Sep 07 22:33:09.724525 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:09.724366    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 07 22:33:09.724525 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:09.724425    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls podName:3cd41b43-6357-4ec5-b4b1-0cfc785ea58b nodeName:}" failed. No retries permitted until 2026-09-07 22:33:11.724408931 +0000 UTC m=+74.781241885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-28hz4" (UID: "3cd41b43-6357-4ec5-b4b1-0cfc785ea58b") : secret "cluster-monitoring-operator-tls" not found
Sep 07 22:33:09.864125 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:09.864086    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-nrwrg" event={"ID":"d3159cd3-b8ca-403e-a8c2-687db53940ab","Type":"ContainerStarted","Data":"62535a314c8802b1322280f7893c95a390b676394ae0b2db44430bcb1d02aeb1"}
Sep 07 22:33:09.865625 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:09.865592    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jd8ww" event={"ID":"1e97b2da-120f-4a72-a6ea-907bcb72a13e","Type":"ContainerStarted","Data":"d167aedee8db2d1adf782223a4a45ec98d1e08a273294fb527ff6f9200b8a107"}
Sep 07 22:33:09.867097 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:09.867074    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q6zr5" event={"ID":"0a7d1434-99e1-4ab9-8b17-08aebf32c93a","Type":"ContainerStarted","Data":"20532a3d16fe3d7865750463e120406951c89ad7006d22cdf16780dbc31ccc96"}
Sep 07 22:33:09.867263 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:09.867195    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:33:09.878880 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:09.878825    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jd8ww" podStartSLOduration=67.837919693 podStartE2EDuration="1m12.878803664s" totalImagesPullingTime="5.040883971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:09.877789343 +0000 UTC m=+72.934622327" watchObservedRunningTime="2026-09-07 22:33:09.878803664 +0000 UTC m=+72.935636633"
Sep 07 22:33:09.890912 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:09.890853    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q6zr5" podStartSLOduration=67.86023363 podStartE2EDuration="1m12.890831108s" totalImagesPullingTime="5.030597478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:09.889872235 +0000 UTC m=+72.946705197" watchObservedRunningTime="2026-09-07 22:33:09.890831108 +0000 UTC m=+72.947664063"
Sep 07 22:33:11.742889 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:11.742769    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-28hz4\" (UID: \"3cd41b43-6357-4ec5-b4b1-0cfc785ea58b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"
Sep 07 22:33:11.742889 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:11.742879    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nzk9w\" (UID: \"e3b9b281-ba77-4b3b-85ff-21e3632ff4ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w"
Sep 07 22:33:11.743372 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:11.742930    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 07 22:33:11.743372 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:11.742998    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls podName:3cd41b43-6357-4ec5-b4b1-0cfc785ea58b nodeName:}" failed. No retries permitted until 2026-09-07 22:33:15.742981753 +0000 UTC m=+78.799814691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-28hz4" (UID: "3cd41b43-6357-4ec5-b4b1-0cfc785ea58b") : secret "cluster-monitoring-operator-tls" not found
Sep 07 22:33:11.743372 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:11.743019    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 07 22:33:11.743372 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:11.743076    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-samples-operator-tls podName:e3b9b281-ba77-4b3b-85ff-21e3632ff4ce nodeName:}" failed. No retries permitted until 2026-09-07 22:33:15.743061254 +0000 UTC m=+78.799894208 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nzk9w" (UID: "e3b9b281-ba77-4b3b-85ff-21e3632ff4ce") : secret "samples-operator-tls" not found
Sep 07 22:33:11.874542 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:11.874515    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/0.log"
Sep 07 22:33:11.874701 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:11.874555    2567 generic.go:356] "Generic (PLEG): container finished" podID="d3159cd3-b8ca-403e-a8c2-687db53940ab" containerID="497606979fd57bf91ff608cfe8fc066fbb9f9209e26829d6393b858eff741e65" exitCode=255
Sep 07 22:33:11.874701 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:11.874584    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-nrwrg" event={"ID":"d3159cd3-b8ca-403e-a8c2-687db53940ab","Type":"ContainerDied","Data":"497606979fd57bf91ff608cfe8fc066fbb9f9209e26829d6393b858eff741e65"}
Sep 07 22:33:11.874887 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:11.874870    2567 scope.go:117] "RemoveContainer" containerID="497606979fd57bf91ff608cfe8fc066fbb9f9209e26829d6393b858eff741e65"
Sep 07 22:33:12.563343 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.563308    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pqgx4"]
Sep 07 22:33:12.566525 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.566502    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:12.568482 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.568459    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 07 22:33:12.568612 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.568495    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 07 22:33:12.569301 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.569269    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 07 22:33:12.569431 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.569364    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 07 22:33:12.569431 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.569371    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tb8g9\""
Sep 07 22:33:12.576588 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.576567    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pqgx4"]
Sep 07 22:33:12.650117 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.650080    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46kw7\" (UniqueName: \"kubernetes.io/projected/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-api-access-46kw7\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:12.650317 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.650147    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db59dacc-3974-4a74-91ae-1b083c32f17a-crio-socket\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:12.650317 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.650187    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/db59dacc-3974-4a74-91ae-1b083c32f17a-data-volume\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:12.650317 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.650289    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:12.650426 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.650316    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:12.751608 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.751557    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46kw7\" (UniqueName: \"kubernetes.io/projected/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-api-access-46kw7\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:12.752001 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.751642    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db59dacc-3974-4a74-91ae-1b083c32f17a-crio-socket\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:12.752001 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.751666    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/db59dacc-3974-4a74-91ae-1b083c32f17a-data-volume\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:12.752001 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.751702    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:12.752001 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.751725    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:12.752001 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.751764    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db59dacc-3974-4a74-91ae-1b083c32f17a-crio-socket\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:12.752001 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:12.751794    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 22:33:12.752001 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:12.751813    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:12.752001 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:12.751864    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-rbac-proxy-cm podName:db59dacc-3974-4a74-91ae-1b083c32f17a nodeName:}" failed. No retries permitted until 2026-09-07 22:33:13.251849317 +0000 UTC m=+76.308682274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pqgx4" (UID: "db59dacc-3974-4a74-91ae-1b083c32f17a") : configmap "kube-rbac-proxy" not found
Sep 07 22:33:12.752001 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:12.751878    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls podName:db59dacc-3974-4a74-91ae-1b083c32f17a nodeName:}" failed. No retries permitted until 2026-09-07 22:33:13.251872033 +0000 UTC m=+76.308704972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pqgx4" (UID: "db59dacc-3974-4a74-91ae-1b083c32f17a") : secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:12.752001 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.751983    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/db59dacc-3974-4a74-91ae-1b083c32f17a-data-volume\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:12.759555 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.759523    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46kw7\" (UniqueName: \"kubernetes.io/projected/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-api-access-46kw7\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:12.879150 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.879123    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/1.log"
Sep 07 22:33:12.879596 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.879578    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/0.log"
Sep 07 22:33:12.879666 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.879617    2567 generic.go:356] "Generic (PLEG): container finished" podID="d3159cd3-b8ca-403e-a8c2-687db53940ab" containerID="2217a806ac9a4ed554a3c81cfec18a491c6cade90bc8254eaaa6133cfc2346f5" exitCode=255
Sep 07 22:33:12.879702 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.879653    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-nrwrg" event={"ID":"d3159cd3-b8ca-403e-a8c2-687db53940ab","Type":"ContainerDied","Data":"2217a806ac9a4ed554a3c81cfec18a491c6cade90bc8254eaaa6133cfc2346f5"}
Sep 07 22:33:12.879740 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.879712    2567 scope.go:117] "RemoveContainer" containerID="497606979fd57bf91ff608cfe8fc066fbb9f9209e26829d6393b858eff741e65"
Sep 07 22:33:12.879926 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:12.879910    2567 scope.go:117] "RemoveContainer" containerID="2217a806ac9a4ed554a3c81cfec18a491c6cade90bc8254eaaa6133cfc2346f5"
Sep 07 22:33:12.880104 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:12.880083    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-nrwrg_openshift-console-operator(d3159cd3-b8ca-403e-a8c2-687db53940ab)\"" pod="openshift-console-operator/console-operator-85b454874b-nrwrg" podUID="d3159cd3-b8ca-403e-a8c2-687db53940ab"
Sep 07 22:33:13.256865 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:13.256769    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:13.256865 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:13.256809    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:13.257068 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:13.256899    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 22:33:13.257068 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:13.256931    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:13.257068 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:13.256977    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-rbac-proxy-cm podName:db59dacc-3974-4a74-91ae-1b083c32f17a nodeName:}" failed. No retries permitted until 2026-09-07 22:33:14.256959646 +0000 UTC m=+77.313792585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pqgx4" (UID: "db59dacc-3974-4a74-91ae-1b083c32f17a") : configmap "kube-rbac-proxy" not found
Sep 07 22:33:13.257068 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:13.256992    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls podName:db59dacc-3974-4a74-91ae-1b083c32f17a nodeName:}" failed. No retries permitted until 2026-09-07 22:33:14.25698587 +0000 UTC m=+77.313818809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pqgx4" (UID: "db59dacc-3974-4a74-91ae-1b083c32f17a") : secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:13.883909 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:13.883882    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/1.log"
Sep 07 22:33:13.884283 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:13.884188    2567 scope.go:117] "RemoveContainer" containerID="2217a806ac9a4ed554a3c81cfec18a491c6cade90bc8254eaaa6133cfc2346f5"
Sep 07 22:33:13.884373 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:13.884352    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-nrwrg_openshift-console-operator(d3159cd3-b8ca-403e-a8c2-687db53940ab)\"" pod="openshift-console-operator/console-operator-85b454874b-nrwrg" podUID="d3159cd3-b8ca-403e-a8c2-687db53940ab"
Sep 07 22:33:14.265712 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:14.265616    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:14.265712 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:14.265659    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:14.265906 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:14.265780    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:14.265906 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:14.265838    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls podName:db59dacc-3974-4a74-91ae-1b083c32f17a nodeName:}" failed. No retries permitted until 2026-09-07 22:33:16.265818377 +0000 UTC m=+79.322651316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pqgx4" (UID: "db59dacc-3974-4a74-91ae-1b083c32f17a") : secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:14.265906 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:14.265848    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 22:33:14.265906 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:14.265899    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-rbac-proxy-cm podName:db59dacc-3974-4a74-91ae-1b083c32f17a nodeName:}" failed. No retries permitted until 2026-09-07 22:33:16.265886041 +0000 UTC m=+79.322718981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pqgx4" (UID: "db59dacc-3974-4a74-91ae-1b083c32f17a") : configmap "kube-rbac-proxy" not found
Sep 07 22:33:14.726357 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:14.726330    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vb75n_b09b22bc-14fb-423b-b7b4-0d3620da9bca/dns-node-resolver/0.log"
Sep 07 22:33:15.327634 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:15.327602    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cz944_56a27e39-3141-4765-8e18-d8195a6c567e/node-ca/0.log"
Sep 07 22:33:15.778882 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:15.778778    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nzk9w\" (UID: \"e3b9b281-ba77-4b3b-85ff-21e3632ff4ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w"
Sep 07 22:33:15.779039 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:15.778889    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-28hz4\" (UID: \"3cd41b43-6357-4ec5-b4b1-0cfc785ea58b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"
Sep 07 22:33:15.779039 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:15.778946    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 07 22:33:15.779039 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:15.778976    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 07 22:33:15.779039 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:15.779026    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls podName:3cd41b43-6357-4ec5-b4b1-0cfc785ea58b nodeName:}" failed. No retries permitted until 2026-09-07 22:33:23.779010696 +0000 UTC m=+86.835843641 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-28hz4" (UID: "3cd41b43-6357-4ec5-b4b1-0cfc785ea58b") : secret "cluster-monitoring-operator-tls" not found
Sep 07 22:33:15.779039 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:15.779041    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-samples-operator-tls podName:e3b9b281-ba77-4b3b-85ff-21e3632ff4ce nodeName:}" failed. No retries permitted until 2026-09-07 22:33:23.779034617 +0000 UTC m=+86.835867556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nzk9w" (UID: "e3b9b281-ba77-4b3b-85ff-21e3632ff4ce") : secret "samples-operator-tls" not found
Sep 07 22:33:16.282952 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:16.282915    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:16.282952 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:16.282955    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:16.283161 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:16.283052    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 22:33:16.283161 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:16.283132    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-rbac-proxy-cm podName:db59dacc-3974-4a74-91ae-1b083c32f17a nodeName:}" failed. No retries permitted until 2026-09-07 22:33:20.28311659 +0000 UTC m=+83.339949529 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pqgx4" (UID: "db59dacc-3974-4a74-91ae-1b083c32f17a") : configmap "kube-rbac-proxy" not found
Sep 07 22:33:16.283161 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:16.283062    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:16.283281 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:16.283186    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls podName:db59dacc-3974-4a74-91ae-1b083c32f17a nodeName:}" failed. No retries permitted until 2026-09-07 22:33:20.283175036 +0000 UTC m=+83.340007975 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pqgx4" (UID: "db59dacc-3974-4a74-91ae-1b083c32f17a") : secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:16.327379 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:16.327345    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lxqwk_a7e100c0-fa8b-4565-978b-f3fa81227e5a/serve-healthcheck-canary/0.log"
Sep 07 22:33:18.209860 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:18.209821    2567 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:18.210276 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:18.209913    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:18.210276 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:18.210225    2567 scope.go:117] "RemoveContainer" containerID="2217a806ac9a4ed554a3c81cfec18a491c6cade90bc8254eaaa6133cfc2346f5"
Sep 07 22:33:18.210416 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:18.210396    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-nrwrg_openshift-console-operator(d3159cd3-b8ca-403e-a8c2-687db53940ab)\"" pod="openshift-console-operator/console-operator-85b454874b-nrwrg" podUID="d3159cd3-b8ca-403e-a8c2-687db53940ab"
Sep 07 22:33:20.318820 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:20.318697    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:20.318820 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:20.318742    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:20.319271 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:20.318821    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 22:33:20.319271 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:20.318836    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:20.319271 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:20.318895    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-rbac-proxy-cm podName:db59dacc-3974-4a74-91ae-1b083c32f17a nodeName:}" failed. No retries permitted until 2026-09-07 22:33:28.318873951 +0000 UTC m=+91.375706906 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pqgx4" (UID: "db59dacc-3974-4a74-91ae-1b083c32f17a") : configmap "kube-rbac-proxy" not found
Sep 07 22:33:20.319271 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:20.318929    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls podName:db59dacc-3974-4a74-91ae-1b083c32f17a nodeName:}" failed. No retries permitted until 2026-09-07 22:33:28.318905539 +0000 UTC m=+91.375738492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pqgx4" (UID: "db59dacc-3974-4a74-91ae-1b083c32f17a") : secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:23.850085 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:23.850040    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-28hz4\" (UID: \"3cd41b43-6357-4ec5-b4b1-0cfc785ea58b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"
Sep 07 22:33:23.850500 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:23.850147    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nzk9w\" (UID: \"e3b9b281-ba77-4b3b-85ff-21e3632ff4ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w"
Sep 07 22:33:23.850500 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:23.850219    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 07 22:33:23.850500 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:23.850304    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls podName:3cd41b43-6357-4ec5-b4b1-0cfc785ea58b nodeName:}" failed. No retries permitted until 2026-09-07 22:33:39.850287421 +0000 UTC m=+102.907120360 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-28hz4" (UID: "3cd41b43-6357-4ec5-b4b1-0cfc785ea58b") : secret "cluster-monitoring-operator-tls" not found
Sep 07 22:33:23.852438 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:23.852420    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3b9b281-ba77-4b3b-85ff-21e3632ff4ce-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nzk9w\" (UID: \"e3b9b281-ba77-4b3b-85ff-21e3632ff4ce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w"
Sep 07 22:33:23.908643 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:23.908613    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w"
Sep 07 22:33:24.175273 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:24.175181    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w"]
Sep 07 22:33:24.912259 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:24.912213    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w" event={"ID":"e3b9b281-ba77-4b3b-85ff-21e3632ff4ce","Type":"ContainerStarted","Data":"23cbca12223b8111830044616a145aec7532faf018cdc5b40b0cbfad6ee49c45"}
Sep 07 22:33:26.921181 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:26.921144    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w" event={"ID":"e3b9b281-ba77-4b3b-85ff-21e3632ff4ce","Type":"ContainerStarted","Data":"147e649cd9cd982736646f3781dfdcc5c436c750a947c67c59cb0d9a55c5d09e"}
Sep 07 22:33:26.921679 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:26.921205    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w" event={"ID":"e3b9b281-ba77-4b3b-85ff-21e3632ff4ce","Type":"ContainerStarted","Data":"5ec5fc4f8758354cdf9ccb3879c6df4668867079b32f0cee008a890d4ab9b177"}
Sep 07 22:33:26.936165 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:26.936112    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nzk9w" podStartSLOduration=18.10260743 podStartE2EDuration="19.93609642s" totalImagesPullingTime="1.83348899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:26.934791615 +0000 UTC m=+89.991624576" watchObservedRunningTime="2026-09-07 22:33:26.93609642 +0000 UTC m=+89.992929380"
Sep 07 22:33:28.392047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:28.391998    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:28.392047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:28.392048    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:28.392699 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:28.392179    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:28.392699 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:28.392275    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls podName:db59dacc-3974-4a74-91ae-1b083c32f17a nodeName:}" failed. No retries permitted until 2026-09-07 22:33:44.392233678 +0000 UTC m=+107.449066624 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pqgx4" (UID: "db59dacc-3974-4a74-91ae-1b083c32f17a") : secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:28.392699 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:28.392527    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db59dacc-3974-4a74-91ae-1b083c32f17a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:31.460115 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:31.460075    2567 scope.go:117] "RemoveContainer" containerID="2217a806ac9a4ed554a3c81cfec18a491c6cade90bc8254eaaa6133cfc2346f5"
Sep 07 22:33:31.937611 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:31.937583    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/2.log"
Sep 07 22:33:31.941497 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:31.941469    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/1.log"
Sep 07 22:33:31.941642 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:31.941514    2567 generic.go:356] "Generic (PLEG): container finished" podID="d3159cd3-b8ca-403e-a8c2-687db53940ab" containerID="6005a6045b7c537db7e3c5d953c06d393f226b736a238b1778f2e105ee58f12d" exitCode=255
Sep 07 22:33:31.941642 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:31.941566    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-nrwrg" event={"ID":"d3159cd3-b8ca-403e-a8c2-687db53940ab","Type":"ContainerDied","Data":"6005a6045b7c537db7e3c5d953c06d393f226b736a238b1778f2e105ee58f12d"}
Sep 07 22:33:31.941748 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:31.941651    2567 scope.go:117] "RemoveContainer" containerID="2217a806ac9a4ed554a3c81cfec18a491c6cade90bc8254eaaa6133cfc2346f5"
Sep 07 22:33:31.941950 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:31.941925    2567 scope.go:117] "RemoveContainer" containerID="6005a6045b7c537db7e3c5d953c06d393f226b736a238b1778f2e105ee58f12d"
Sep 07 22:33:31.942125 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:31.942105    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-nrwrg_openshift-console-operator(d3159cd3-b8ca-403e-a8c2-687db53940ab)\"" pod="openshift-console-operator/console-operator-85b454874b-nrwrg" podUID="d3159cd3-b8ca-403e-a8c2-687db53940ab"
Sep 07 22:33:32.946179 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:32.946152    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/2.log"
Sep 07 22:33:35.251220 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:35.251176    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:33:35.251595 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:35.251268    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls\") pod \"dns-default-cwg2q\" (UID: \"90c9c42a-571f-4e9c-a741-c4ac016b1308\") " pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:33:35.253613 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:35.253578    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90c9c42a-571f-4e9c-a741-c4ac016b1308-metrics-tls\") pod \"dns-default-cwg2q\" (UID: \"90c9c42a-571f-4e9c-a741-c4ac016b1308\") " pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:33:35.253757 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:35.253737    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls\") pod \"image-registry-79998dbc84-kskwc\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") " pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:33:35.501432 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:35.501361    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hvdml\""
Sep 07 22:33:35.510321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:35.510302    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:33:35.533389 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:35.533361    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zd2nk\""
Sep 07 22:33:35.541935 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:35.541912    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:33:35.783686 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:35.783603    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79998dbc84-kskwc"]
Sep 07 22:33:35.787786 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:33:35.787758    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6836bd00_5ba9_4802_b393_0d830fb1eba1.slice/crio-e683c3c2736f9b26fb2a401661cf21920136962e5a5618a16cb266430b306a74 WatchSource:0}: Error finding container e683c3c2736f9b26fb2a401661cf21920136962e5a5618a16cb266430b306a74: Status 404 returned error can't find the container with id e683c3c2736f9b26fb2a401661cf21920136962e5a5618a16cb266430b306a74
Sep 07 22:33:35.817536 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:35.817507    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cwg2q"]
Sep 07 22:33:35.820741 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:33:35.820715    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c9c42a_571f_4e9c_a741_c4ac016b1308.slice/crio-bc5d71c49d6670f38fa87470e36efee7841b1c6b4e40f6c5859bf61f191b47aa WatchSource:0}: Error finding container bc5d71c49d6670f38fa87470e36efee7841b1c6b4e40f6c5859bf61f191b47aa: Status 404 returned error can't find the container with id bc5d71c49d6670f38fa87470e36efee7841b1c6b4e40f6c5859bf61f191b47aa
Sep 07 22:33:35.958196 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:35.958162    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79998dbc84-kskwc" event={"ID":"6836bd00-5ba9-4802-b393-0d830fb1eba1","Type":"ContainerStarted","Data":"4cbef3bc2163ec3d526a610dc7e87f60a56760c1844032e9f0edbf1a60f64bc7"}
Sep 07 22:33:35.958385 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:35.958219    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79998dbc84-kskwc" event={"ID":"6836bd00-5ba9-4802-b393-0d830fb1eba1","Type":"ContainerStarted","Data":"e683c3c2736f9b26fb2a401661cf21920136962e5a5618a16cb266430b306a74"}
Sep 07 22:33:35.958385 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:35.958261    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:33:35.959607 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:35.959575    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cwg2q" event={"ID":"90c9c42a-571f-4e9c-a741-c4ac016b1308","Type":"ContainerStarted","Data":"bc5d71c49d6670f38fa87470e36efee7841b1c6b4e40f6c5859bf61f191b47aa"}
Sep 07 22:33:35.976937 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:35.976881    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79998dbc84-kskwc" podStartSLOduration=98.976861635 podStartE2EDuration="1m38.976861635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:35.975932729 +0000 UTC m=+99.032765710" watchObservedRunningTime="2026-09-07 22:33:35.976861635 +0000 UTC m=+99.033694595"
Sep 07 22:33:37.351376 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.351343    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55bdb96b98-wp72t"]
Sep 07 22:33:37.354739 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.354716    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55bdb96b98-wp72t"
Sep 07 22:33:37.360278 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.359575    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 07 22:33:37.360278 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.359600    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 07 22:33:37.360278 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.360162    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 07 22:33:37.360536 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.360382    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 07 22:33:37.360536 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.360478    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-p6lcs\""
Sep 07 22:33:37.367787 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.367766    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55bdb96b98-wp72t"]
Sep 07 22:33:37.469269 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.469226    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/94f23605-0c18-4a80-acfb-4e8d8b6ba6dc-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55bdb96b98-wp72t\" (UID: \"94f23605-0c18-4a80-acfb-4e8d8b6ba6dc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55bdb96b98-wp72t"
Sep 07 22:33:37.469398 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.469283    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvdlq\" (UniqueName: \"kubernetes.io/projected/94f23605-0c18-4a80-acfb-4e8d8b6ba6dc-kube-api-access-xvdlq\") pod \"managed-serviceaccount-addon-agent-55bdb96b98-wp72t\" (UID: \"94f23605-0c18-4a80-acfb-4e8d8b6ba6dc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55bdb96b98-wp72t"
Sep 07 22:33:37.569868 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.569836    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/94f23605-0c18-4a80-acfb-4e8d8b6ba6dc-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55bdb96b98-wp72t\" (UID: \"94f23605-0c18-4a80-acfb-4e8d8b6ba6dc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55bdb96b98-wp72t"
Sep 07 22:33:37.570022 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.569876    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvdlq\" (UniqueName: \"kubernetes.io/projected/94f23605-0c18-4a80-acfb-4e8d8b6ba6dc-kube-api-access-xvdlq\") pod \"managed-serviceaccount-addon-agent-55bdb96b98-wp72t\" (UID: \"94f23605-0c18-4a80-acfb-4e8d8b6ba6dc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55bdb96b98-wp72t"
Sep 07 22:33:37.572219 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.572197    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/94f23605-0c18-4a80-acfb-4e8d8b6ba6dc-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55bdb96b98-wp72t\" (UID: \"94f23605-0c18-4a80-acfb-4e8d8b6ba6dc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55bdb96b98-wp72t"
Sep 07 22:33:37.577653 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.577626    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvdlq\" (UniqueName: \"kubernetes.io/projected/94f23605-0c18-4a80-acfb-4e8d8b6ba6dc-kube-api-access-xvdlq\") pod \"managed-serviceaccount-addon-agent-55bdb96b98-wp72t\" (UID: \"94f23605-0c18-4a80-acfb-4e8d8b6ba6dc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55bdb96b98-wp72t"
Sep 07 22:33:37.677341 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.677307    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55bdb96b98-wp72t"
Sep 07 22:33:37.946716 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.946639    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55bdb96b98-wp72t"]
Sep 07 22:33:37.949731 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:33:37.949700    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94f23605_0c18_4a80_acfb_4e8d8b6ba6dc.slice/crio-89ec72275bc300cda2ebfd9601b1c5e58dce4abcda98fa880abd66a470f004f3 WatchSource:0}: Error finding container 89ec72275bc300cda2ebfd9601b1c5e58dce4abcda98fa880abd66a470f004f3: Status 404 returned error can't find the container with id 89ec72275bc300cda2ebfd9601b1c5e58dce4abcda98fa880abd66a470f004f3
Sep 07 22:33:37.966863 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.966831    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55bdb96b98-wp72t" event={"ID":"94f23605-0c18-4a80-acfb-4e8d8b6ba6dc","Type":"ContainerStarted","Data":"89ec72275bc300cda2ebfd9601b1c5e58dce4abcda98fa880abd66a470f004f3"}
Sep 07 22:33:37.968515 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.968486    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cwg2q" event={"ID":"90c9c42a-571f-4e9c-a741-c4ac016b1308","Type":"ContainerStarted","Data":"6284142997d38e8f081a43b75918799053eb76838e7cb0ff84b26e0950c97584"}
Sep 07 22:33:37.968647 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.968536    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cwg2q" event={"ID":"90c9c42a-571f-4e9c-a741-c4ac016b1308","Type":"ContainerStarted","Data":"5befc60a7ff237329ed5434f1dd268d4570265f15a36b06a4421d1e8042b412d"}
Sep 07 22:33:37.968749 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.968732    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:33:37.984461 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:37.984412    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cwg2q" podStartSLOduration=65.394922974 podStartE2EDuration="1m6.984393489s" totalImagesPullingTime="1.589470515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:37.982913578 +0000 UTC m=+101.039746538" watchObservedRunningTime="2026-09-07 22:33:37.984393489 +0000 UTC m=+101.041226482"
Sep 07 22:33:38.210184 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:38.210100    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:38.210184 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:38.210162    2567 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:38.210505 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:38.210492    2567 scope.go:117] "RemoveContainer" containerID="6005a6045b7c537db7e3c5d953c06d393f226b736a238b1778f2e105ee58f12d"
Sep 07 22:33:38.210672 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:38.210653    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-nrwrg_openshift-console-operator(d3159cd3-b8ca-403e-a8c2-687db53940ab)\"" pod="openshift-console-operator/console-operator-85b454874b-nrwrg" podUID="d3159cd3-b8ca-403e-a8c2-687db53940ab"
Sep 07 22:33:39.888920 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:39.888858    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-28hz4\" (UID: \"3cd41b43-6357-4ec5-b4b1-0cfc785ea58b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"
Sep 07 22:33:39.891708 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:39.891678    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cd41b43-6357-4ec5-b4b1-0cfc785ea58b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-28hz4\" (UID: \"3cd41b43-6357-4ec5-b4b1-0cfc785ea58b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"
Sep 07 22:33:40.101857 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:40.101819    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"
Sep 07 22:33:40.670225 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:40.670150    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4"]
Sep 07 22:33:40.696837 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:33:40.696796    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd41b43_6357_4ec5_b4b1_0cfc785ea58b.slice/crio-8c8dc7328e816aadfcf6d8a4f95ef324f127badca4ddaed3154f2c095d588cd8 WatchSource:0}: Error finding container 8c8dc7328e816aadfcf6d8a4f95ef324f127badca4ddaed3154f2c095d588cd8: Status 404 returned error can't find the container with id 8c8dc7328e816aadfcf6d8a4f95ef324f127badca4ddaed3154f2c095d588cd8
Sep 07 22:33:40.872719 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:40.872689    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q6zr5"
Sep 07 22:33:40.984657 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:40.984563    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55bdb96b98-wp72t" event={"ID":"94f23605-0c18-4a80-acfb-4e8d8b6ba6dc","Type":"ContainerStarted","Data":"1644d19e154cbd0a25783a62d2ed9236011e93990afd1dc7f1c855c5d5da2b54"}
Sep 07 22:33:40.985817 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:40.985788    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4" event={"ID":"3cd41b43-6357-4ec5-b4b1-0cfc785ea58b","Type":"ContainerStarted","Data":"8c8dc7328e816aadfcf6d8a4f95ef324f127badca4ddaed3154f2c095d588cd8"}
Sep 07 22:33:40.996742 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:40.996689    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55bdb96b98-wp72t" podStartSLOduration=1.231502239 podStartE2EDuration="3.996670584s" totalImagesPullingTime="2.765168345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:40.996391831 +0000 UTC m=+104.053224793" watchObservedRunningTime="2026-09-07 22:33:40.996670584 +0000 UTC m=+104.053503527"
Sep 07 22:33:43.996421 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:43.996385    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4" event={"ID":"3cd41b43-6357-4ec5-b4b1-0cfc785ea58b","Type":"ContainerStarted","Data":"e717b7bce0cecab8e778b4c79f4fe73114fd77486a647c94f06871ab63fedbc5"}
Sep 07 22:33:44.010855 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:44.010800    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-28hz4" podStartSLOduration=34.703536713 podStartE2EDuration="37.010778532s" totalImagesPullingTime="2.307241819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:44.009536361 +0000 UTC m=+107.066369323" watchObservedRunningTime="2026-09-07 22:33:44.010778532 +0000 UTC m=+107.067611497"
Sep 07 22:33:44.422203 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:44.422155    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:44.424587 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:44.424558    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db59dacc-3974-4a74-91ae-1b083c32f17a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pqgx4\" (UID: \"db59dacc-3974-4a74-91ae-1b083c32f17a\") " pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:44.676133 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:44.676030    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pqgx4"
Sep 07 22:33:44.942605 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:44.942526    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pqgx4"]
Sep 07 22:33:44.945736 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:33:44.945707    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb59dacc_3974_4a74_91ae_1b083c32f17a.slice/crio-fa0469c7d3e43e5d31d7573f38c45dc029ab2c8667cba734d07ccafb80bf697e WatchSource:0}: Error finding container fa0469c7d3e43e5d31d7573f38c45dc029ab2c8667cba734d07ccafb80bf697e: Status 404 returned error can't find the container with id fa0469c7d3e43e5d31d7573f38c45dc029ab2c8667cba734d07ccafb80bf697e
Sep 07 22:33:45.000901 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:45.000872    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pqgx4" event={"ID":"db59dacc-3974-4a74-91ae-1b083c32f17a","Type":"ContainerStarted","Data":"fa0469c7d3e43e5d31d7573f38c45dc029ab2c8667cba734d07ccafb80bf697e"}
Sep 07 22:33:46.006967 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:46.006930    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pqgx4" event={"ID":"db59dacc-3974-4a74-91ae-1b083c32f17a","Type":"ContainerStarted","Data":"012ed55b6354214f33a7bbf7e0abc38a76ecee73ee1e6839d14d260aca2f9f1e"}
Sep 07 22:33:46.007459 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:46.006993    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pqgx4" event={"ID":"db59dacc-3974-4a74-91ae-1b083c32f17a","Type":"ContainerStarted","Data":"0e334462246ebaab1dd4aa42cc458073877e97ea1abe82e4364cb6e208447411"}
Sep 07 22:33:47.015997 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.012731    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pqgx4" event={"ID":"db59dacc-3974-4a74-91ae-1b083c32f17a","Type":"ContainerStarted","Data":"c5d31a22353daf99c6296c46d61650ef5fb764caeca5f937fa9fc88746602f2b"}
Sep 07 22:33:47.029938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.029882    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pqgx4" podStartSLOduration=33.216641005 podStartE2EDuration="35.029864879s" totalImagesPullingTime="1.813223874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:33:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:47.029107194 +0000 UTC m=+110.085940186" watchObservedRunningTime="2026-09-07 22:33:47.029864879 +0000 UTC m=+110.086697840"
Sep 07 22:33:47.588267 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.588218    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-vjxnd"]
Sep 07 22:33:47.591212 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.591195    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-vjxnd"
Sep 07 22:33:47.593415 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.593394    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 07 22:33:47.594321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.594297    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 07 22:33:47.594321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.594312    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-tt7rh\""
Sep 07 22:33:47.594474 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.594353    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 07 22:33:47.601351 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.601325    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-vjxnd"]
Sep 07 22:33:47.644622 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.644583    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f563137-3790-4f41-ae58-98dbe5b37caa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-vjxnd\" (UID: \"6f563137-3790-4f41-ae58-98dbe5b37caa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-vjxnd"
Sep 07 22:33:47.644622 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.644623    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6rbl\" (UniqueName: \"kubernetes.io/projected/6f563137-3790-4f41-ae58-98dbe5b37caa-kube-api-access-x6rbl\") pod \"prometheus-operator-6f4946db56-vjxnd\" (UID: \"6f563137-3790-4f41-ae58-98dbe5b37caa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-vjxnd"
Sep 07 22:33:47.644851 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.644643    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f563137-3790-4f41-ae58-98dbe5b37caa-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-vjxnd\" (UID: \"6f563137-3790-4f41-ae58-98dbe5b37caa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-vjxnd"
Sep 07 22:33:47.644851 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.644727    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f563137-3790-4f41-ae58-98dbe5b37caa-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-vjxnd\" (UID: \"6f563137-3790-4f41-ae58-98dbe5b37caa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-vjxnd"
Sep 07 22:33:47.745555 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.745513    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f563137-3790-4f41-ae58-98dbe5b37caa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-vjxnd\" (UID: \"6f563137-3790-4f41-ae58-98dbe5b37caa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-vjxnd"
Sep 07 22:33:47.745555 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.745553    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6rbl\" (UniqueName: \"kubernetes.io/projected/6f563137-3790-4f41-ae58-98dbe5b37caa-kube-api-access-x6rbl\") pod \"prometheus-operator-6f4946db56-vjxnd\" (UID: \"6f563137-3790-4f41-ae58-98dbe5b37caa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-vjxnd"
Sep 07 22:33:47.745816 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.745578    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f563137-3790-4f41-ae58-98dbe5b37caa-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-vjxnd\" (UID: \"6f563137-3790-4f41-ae58-98dbe5b37caa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-vjxnd"
Sep 07 22:33:47.745816 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.745611    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f563137-3790-4f41-ae58-98dbe5b37caa-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-vjxnd\" (UID: \"6f563137-3790-4f41-ae58-98dbe5b37caa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-vjxnd"
Sep 07 22:33:47.745816 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:47.745737    2567 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 07 22:33:47.745816 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:47.745817    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f563137-3790-4f41-ae58-98dbe5b37caa-prometheus-operator-tls podName:6f563137-3790-4f41-ae58-98dbe5b37caa nodeName:}" failed. No retries permitted until 2026-09-07 22:33:48.24579709 +0000 UTC m=+111.302630044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/6f563137-3790-4f41-ae58-98dbe5b37caa-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-vjxnd" (UID: "6f563137-3790-4f41-ae58-98dbe5b37caa") : secret "prometheus-operator-tls" not found
Sep 07 22:33:47.746338 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.746316    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f563137-3790-4f41-ae58-98dbe5b37caa-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-vjxnd\" (UID: \"6f563137-3790-4f41-ae58-98dbe5b37caa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-vjxnd"
Sep 07 22:33:47.748053 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.748020    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f563137-3790-4f41-ae58-98dbe5b37caa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-vjxnd\" (UID: \"6f563137-3790-4f41-ae58-98dbe5b37caa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-vjxnd"
Sep 07 22:33:47.756508 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.756473    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6rbl\" (UniqueName: \"kubernetes.io/projected/6f563137-3790-4f41-ae58-98dbe5b37caa-kube-api-access-x6rbl\") pod \"prometheus-operator-6f4946db56-vjxnd\" (UID: \"6f563137-3790-4f41-ae58-98dbe5b37caa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-vjxnd"
Sep 07 22:33:47.974688 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:47.974609    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cwg2q"
Sep 07 22:33:48.249098 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:48.249004    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f563137-3790-4f41-ae58-98dbe5b37caa-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-vjxnd\" (UID: \"6f563137-3790-4f41-ae58-98dbe5b37caa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-vjxnd"
Sep 07 22:33:48.251459 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:48.251436    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f563137-3790-4f41-ae58-98dbe5b37caa-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-vjxnd\" (UID: \"6f563137-3790-4f41-ae58-98dbe5b37caa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-vjxnd"
Sep 07 22:33:48.500222 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:48.500135    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-vjxnd"
Sep 07 22:33:48.768857 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:48.768768    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-vjxnd"]
Sep 07 22:33:48.771505 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:33:48.771471    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f563137_3790_4f41_ae58_98dbe5b37caa.slice/crio-1a3f6ce6bedfe860450037e8299b78f32a9cbebfed25ac12aae642fad5eaf171 WatchSource:0}: Error finding container 1a3f6ce6bedfe860450037e8299b78f32a9cbebfed25ac12aae642fad5eaf171: Status 404 returned error can't find the container with id 1a3f6ce6bedfe860450037e8299b78f32a9cbebfed25ac12aae642fad5eaf171
Sep 07 22:33:49.021400 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:49.021306    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-vjxnd" event={"ID":"6f563137-3790-4f41-ae58-98dbe5b37caa","Type":"ContainerStarted","Data":"1a3f6ce6bedfe860450037e8299b78f32a9cbebfed25ac12aae642fad5eaf171"}
Sep 07 22:33:51.029949 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:51.029908    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-vjxnd" event={"ID":"6f563137-3790-4f41-ae58-98dbe5b37caa","Type":"ContainerStarted","Data":"6a5bd04baa1635dc0e21a8ac10912360c032f217b5cb37365bfbc7cb6fe2501a"}
Sep 07 22:33:51.030358 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:51.029998    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-vjxnd" event={"ID":"6f563137-3790-4f41-ae58-98dbe5b37caa","Type":"ContainerStarted","Data":"e23f35343ab517bc7f92a54a82a236c8caeb04fd2c2bc5aae8dc5527e20e9ef5"}
Sep 07 22:33:51.045018 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:51.044966    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-vjxnd" podStartSLOduration=2.571242437 podStartE2EDuration="4.044931259s" totalImagesPullingTime="1.473688822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:51.043884363 +0000 UTC m=+114.100717326" watchObservedRunningTime="2026-09-07 22:33:51.044931259 +0000 UTC m=+114.101764220"
Sep 07 22:33:52.937563 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:52.937528    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4kjc5"]
Sep 07 22:33:52.940873 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:52.940851    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:52.943273 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:52.943232    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 07 22:33:52.943418 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:52.943322    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 07 22:33:52.943418 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:52.943338    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 07 22:33:52.943418 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:52.943329    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dzc2k\""
Sep 07 22:33:52.984939 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:52.984905    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-root\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:52.984939 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:52.984941    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-node-exporter-tls\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:52.985137 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:52.984960    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-node-exporter-accelerators-collector-config\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:52.985137 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:52.984984    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-sys\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:52.985137 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:52.985074    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cprsj\" (UniqueName: \"kubernetes.io/projected/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-kube-api-access-cprsj\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:52.985137 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:52.985119    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:52.985346 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:52.985148    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-node-exporter-wtmp\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:52.985346 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:52.985164    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-metrics-client-ca\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:52.985346 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:52.985183    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-node-exporter-textfile\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.086498 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.086453    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-root\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.086498 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.086498    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-node-exporter-tls\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.086766 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.086525    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-node-exporter-accelerators-collector-config\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.086766 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.086562    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-sys\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.086766 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.086597    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cprsj\" (UniqueName: \"kubernetes.io/projected/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-kube-api-access-cprsj\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.086766 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:53.086623    2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 07 22:33:53.086766 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:33:53.086697    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-node-exporter-tls podName:1c93c4ac-72e8-454d-9c80-9a19a4efbcfd nodeName:}" failed. No retries permitted until 2026-09-07 22:33:53.58668105 +0000 UTC m=+116.643513992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-node-exporter-tls") pod "node-exporter-4kjc5" (UID: "1c93c4ac-72e8-454d-9c80-9a19a4efbcfd") : secret "node-exporter-tls" not found
Sep 07 22:33:53.086766 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.086711    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-root\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.086766 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.086767    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-sys\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.087107 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.086811    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.087107 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.086858    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-node-exporter-wtmp\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.087107 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.086885    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-metrics-client-ca\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.087107 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.086933    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-node-exporter-textfile\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.087107 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.087011    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-node-exporter-wtmp\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.087431 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.087234    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-node-exporter-accelerators-collector-config\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.087431 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.087277    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-node-exporter-textfile\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.087431 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.087398    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-metrics-client-ca\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.089311 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.089289    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.098293 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.098269    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cprsj\" (UniqueName: \"kubernetes.io/projected/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-kube-api-access-cprsj\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.459797 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.459765    2567 scope.go:117] "RemoveContainer" containerID="6005a6045b7c537db7e3c5d953c06d393f226b736a238b1778f2e105ee58f12d"
Sep 07 22:33:53.592012 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.591973    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-node-exporter-tls\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.594473 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.594447    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1c93c4ac-72e8-454d-9c80-9a19a4efbcfd-node-exporter-tls\") pod \"node-exporter-4kjc5\" (UID: \"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd\") " pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.850170 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.850128    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4kjc5"
Sep 07 22:33:53.862044 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:33:53.862011    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c93c4ac_72e8_454d_9c80_9a19a4efbcfd.slice/crio-8ecc246f64b2d199ac2cb78d8d67d3bb36c793b4843e281cc266f34d73ee2615 WatchSource:0}: Error finding container 8ecc246f64b2d199ac2cb78d8d67d3bb36c793b4843e281cc266f34d73ee2615: Status 404 returned error can't find the container with id 8ecc246f64b2d199ac2cb78d8d67d3bb36c793b4843e281cc266f34d73ee2615
Sep 07 22:33:53.991666 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.991628    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 22:33:53.996333 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:53.996303    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.000519 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.000480    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 07 22:33:54.000682 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.000480    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 07 22:33:54.000830 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.000812    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 07 22:33:54.001501 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.001161    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 07 22:33:54.001501 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.001280    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 07 22:33:54.001501 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.001282    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 07 22:33:54.001501 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.001313    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 07 22:33:54.001501 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.001403    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qllvm\""
Sep 07 22:33:54.001793 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.001532    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 07 22:33:54.001793 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.001606    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 07 22:33:54.014845 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.010154    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 22:33:54.041175 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.041125    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4kjc5" event={"ID":"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd","Type":"ContainerStarted","Data":"8ecc246f64b2d199ac2cb78d8d67d3bb36c793b4843e281cc266f34d73ee2615"}
Sep 07 22:33:54.045013 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.044980    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/2.log"
Sep 07 22:33:54.045199 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.045079    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-nrwrg" event={"ID":"d3159cd3-b8ca-403e-a8c2-687db53940ab","Type":"ContainerStarted","Data":"50026752a6e75b08e07cc041ba61463ec2ea59ea004dd7171df14d6781ac5dc6"}
Sep 07 22:33:54.045493 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.045469    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:54.063223 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.063122    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-nrwrg" podStartSLOduration=44.903675848 podStartE2EDuration="47.063100434s" totalImagesPullingTime="2.159424586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:54.062325964 +0000 UTC m=+117.119158926" watchObservedRunningTime="2026-09-07 22:33:54.063100434 +0000 UTC m=+117.119933397"
Sep 07 22:33:54.096605 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.096562    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.096766 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.096617    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.096766 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.096700    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.096873 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.096761    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-web-config\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.096873 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.096786    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc025f8-0b6c-46b4-a44a-a40e35a8035a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.096974 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.096910    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efc025f8-0b6c-46b4-a44a-a40e35a8035a-config-out\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.096974 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.096947    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq7dc\" (UniqueName: \"kubernetes.io/projected/efc025f8-0b6c-46b4-a44a-a40e35a8035a-kube-api-access-tq7dc\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.097079 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.096971    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/efc025f8-0b6c-46b4-a44a-a40e35a8035a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.097079 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.096997    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-config-volume\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.097079 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.097023    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efc025f8-0b6c-46b4-a44a-a40e35a8035a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.097079 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.097051    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.097294 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.097104    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.097294 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.097180    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efc025f8-0b6c-46b4-a44a-a40e35a8035a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.197958 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.197844    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.197958 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.197890    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.197958 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.197921    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.197958 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.197956    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-web-config\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.198311 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.197980    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc025f8-0b6c-46b4-a44a-a40e35a8035a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.198311 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.198012    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efc025f8-0b6c-46b4-a44a-a40e35a8035a-config-out\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.198311 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.198040    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq7dc\" (UniqueName: \"kubernetes.io/projected/efc025f8-0b6c-46b4-a44a-a40e35a8035a-kube-api-access-tq7dc\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.198311 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.198069    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/efc025f8-0b6c-46b4-a44a-a40e35a8035a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.198511 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.198345    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-config-volume\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.198511 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.198393    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efc025f8-0b6c-46b4-a44a-a40e35a8035a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.198511 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.198427    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.198511 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.198459    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.199206 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.198888    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/efc025f8-0b6c-46b4-a44a-a40e35a8035a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.199441 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.199413    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc025f8-0b6c-46b4-a44a-a40e35a8035a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.199592 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.199425    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efc025f8-0b6c-46b4-a44a-a40e35a8035a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.200402 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.200376    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efc025f8-0b6c-46b4-a44a-a40e35a8035a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.201279 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.201210    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.201389 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.201362    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.202232 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.202174    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.202351 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.202329    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-config-volume\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.203443 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.203418    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efc025f8-0b6c-46b4-a44a-a40e35a8035a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.203629 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.203599    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efc025f8-0b6c-46b4-a44a-a40e35a8035a-config-out\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.203982 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.203958    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.204050 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.204029    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.204528 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.204497    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-web-config\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.206951 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.206926    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq7dc\" (UniqueName: \"kubernetes.io/projected/efc025f8-0b6c-46b4-a44a-a40e35a8035a-kube-api-access-tq7dc\") pod \"alertmanager-main-0\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.310378 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.310334    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:33:54.453861 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.453789    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-nrwrg"
Sep 07 22:33:54.602352 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.602316    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 22:33:54.607196 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:33:54.607152    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc025f8_0b6c_46b4_a44a_a40e35a8035a.slice/crio-66b62eeef12acb854d757225321b3827a93ee0791ec42bb210662e2a83d0c7b4 WatchSource:0}: Error finding container 66b62eeef12acb854d757225321b3827a93ee0791ec42bb210662e2a83d0c7b4: Status 404 returned error can't find the container with id 66b62eeef12acb854d757225321b3827a93ee0791ec42bb210662e2a83d0c7b4
Sep 07 22:33:54.624979 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.624940    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-d6b77"]
Sep 07 22:33:54.628670 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.628643    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-d6b77"
Sep 07 22:33:54.631112 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.631085    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-rlwql\""
Sep 07 22:33:54.631259 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.631113    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 07 22:33:54.631259 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.631086    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 07 22:33:54.636279 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.636235    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-d6b77"]
Sep 07 22:33:54.704989 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.704895    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb5v7\" (UniqueName: \"kubernetes.io/projected/87bdbb8b-6a3b-4a92-a68e-3ead40f9d615-kube-api-access-bb5v7\") pod \"downloads-85d98c998f-d6b77\" (UID: \"87bdbb8b-6a3b-4a92-a68e-3ead40f9d615\") " pod="openshift-console/downloads-85d98c998f-d6b77"
Sep 07 22:33:54.805530 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.805498    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb5v7\" (UniqueName: \"kubernetes.io/projected/87bdbb8b-6a3b-4a92-a68e-3ead40f9d615-kube-api-access-bb5v7\") pod \"downloads-85d98c998f-d6b77\" (UID: \"87bdbb8b-6a3b-4a92-a68e-3ead40f9d615\") " pod="openshift-console/downloads-85d98c998f-d6b77"
Sep 07 22:33:54.813737 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.813705    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb5v7\" (UniqueName: \"kubernetes.io/projected/87bdbb8b-6a3b-4a92-a68e-3ead40f9d615-kube-api-access-bb5v7\") pod \"downloads-85d98c998f-d6b77\" (UID: \"87bdbb8b-6a3b-4a92-a68e-3ead40f9d615\") " pod="openshift-console/downloads-85d98c998f-d6b77"
Sep 07 22:33:54.941001 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:54.940967    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-d6b77"
Sep 07 22:33:55.050406 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:55.050352    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4kjc5" event={"ID":"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd","Type":"ContainerStarted","Data":"7cc1c525db9a142fc9aa61d7caa63928bf820be88c30dff0645ed90821a26188"}
Sep 07 22:33:55.051607 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:55.051581    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efc025f8-0b6c-46b4-a44a-a40e35a8035a","Type":"ContainerStarted","Data":"66b62eeef12acb854d757225321b3827a93ee0791ec42bb210662e2a83d0c7b4"}
Sep 07 22:33:55.238146 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:55.238055    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-d6b77"]
Sep 07 22:33:55.242202 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:33:55.242157    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87bdbb8b_6a3b_4a92_a68e_3ead40f9d615.slice/crio-64aad73166592a6c744298643a61a2a222763c8d334b8506546a8df331e8963d WatchSource:0}: Error finding container 64aad73166592a6c744298643a61a2a222763c8d334b8506546a8df331e8963d: Status 404 returned error can't find the container with id 64aad73166592a6c744298643a61a2a222763c8d334b8506546a8df331e8963d
Sep 07 22:33:55.515284 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:55.515185    2567 patch_prober.go:28] interesting pod/image-registry-79998dbc84-kskwc container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 07 22:33:55.515284 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:55.515267    2567 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-79998dbc84-kskwc" podUID="6836bd00-5ba9-4802-b393-0d830fb1eba1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 22:33:56.059814 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:56.059611    2567 generic.go:356] "Generic (PLEG): container finished" podID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerID="ef565aa0a6949fce93e2da8650371f6f25f41550696185376a14a46256cfb94c" exitCode=0
Sep 07 22:33:56.059814 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:56.059748    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efc025f8-0b6c-46b4-a44a-a40e35a8035a","Type":"ContainerDied","Data":"ef565aa0a6949fce93e2da8650371f6f25f41550696185376a14a46256cfb94c"}
Sep 07 22:33:56.061851 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:56.061822    2567 generic.go:356] "Generic (PLEG): container finished" podID="1c93c4ac-72e8-454d-9c80-9a19a4efbcfd" containerID="7cc1c525db9a142fc9aa61d7caa63928bf820be88c30dff0645ed90821a26188" exitCode=0
Sep 07 22:33:56.061970 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:56.061920    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4kjc5" event={"ID":"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd","Type":"ContainerDied","Data":"7cc1c525db9a142fc9aa61d7caa63928bf820be88c30dff0645ed90821a26188"}
Sep 07 22:33:56.063652 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:56.063521    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-d6b77" event={"ID":"87bdbb8b-6a3b-4a92-a68e-3ead40f9d615","Type":"ContainerStarted","Data":"64aad73166592a6c744298643a61a2a222763c8d334b8506546a8df331e8963d"}
Sep 07 22:33:56.968687 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:56.968653    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:33:57.071589 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.071541    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4kjc5" event={"ID":"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd","Type":"ContainerStarted","Data":"1f80e0b2de3f1d9a45b50c36219f372888b2571845b175f5b661c13b3f852055"}
Sep 07 22:33:57.072135 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.071607    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4kjc5" event={"ID":"1c93c4ac-72e8-454d-9c80-9a19a4efbcfd","Type":"ContainerStarted","Data":"d70c053399f457281b948a1d9c5fe1bbf63cf3ac73223463dd34d2fe41316bf4"}
Sep 07 22:33:57.087605 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.087178    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4kjc5" podStartSLOduration=3.907430616 podStartE2EDuration="5.087156883s" totalImagesPullingTime="1.112428861s" totalInitContainerRuntime="67.297406ms" isStatefulPod=true podCreationTimestamp="2026-09-07 22:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:57.08687965 +0000 UTC m=+120.143712636" watchObservedRunningTime="2026-09-07 22:33:57.087156883 +0000 UTC m=+120.143989844"
Sep 07 22:33:57.242955 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.242864    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5f858c696c-n9mdv"]
Sep 07 22:33:57.246848 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.246716    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.249396 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.249367    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 07 22:33:57.249536 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.249367    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 07 22:33:57.249536 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.249397    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4etaqcibi6g37\""
Sep 07 22:33:57.249536 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.249473    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rwx6r\""
Sep 07 22:33:57.249696 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.249523    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 07 22:33:57.249696 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.249654    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 07 22:33:57.254797 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.254772    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f858c696c-n9mdv"]
Sep 07 22:33:57.329212 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.329173    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55kfl\" (UniqueName: \"kubernetes.io/projected/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-kube-api-access-55kfl\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.329435 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.329228    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-secret-metrics-server-client-certs\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.329435 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.329277    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-secret-metrics-server-tls\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.329435 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.329335    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-audit-log\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.329435 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.329388    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-metrics-server-audit-profiles\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.329673 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.329465    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.329673 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.329538    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-client-ca-bundle\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.430561 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.430521    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-audit-log\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.430711 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.430578    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-metrics-server-audit-profiles\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.430711 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.430616    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.430711 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.430658    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-client-ca-bundle\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.430888 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.430720    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55kfl\" (UniqueName: \"kubernetes.io/projected/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-kube-api-access-55kfl\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.430888 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.430796    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-secret-metrics-server-client-certs\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.430888 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.430840    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-secret-metrics-server-tls\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.431037 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.430981    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-audit-log\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.433223 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.433196    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 07 22:33:57.433363 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.433333    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 07 22:33:57.433471 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.433448    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4etaqcibi6g37\""
Sep 07 22:33:57.433543 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.433498    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 07 22:33:57.433543 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.433517    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 07 22:33:57.439208 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.439176    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55kfl\" (UniqueName: \"kubernetes.io/projected/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-kube-api-access-55kfl\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.442189 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.441879    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-metrics-server-audit-profiles\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.442189 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.441895    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.444909 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.444860    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-secret-metrics-server-tls\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.444909 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.444876    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-secret-metrics-server-client-certs\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.445536 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.445502    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf-client-ca-bundle\") pod \"metrics-server-5f858c696c-n9mdv\" (UID: \"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf\") " pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.562870 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.562783    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rwx6r\""
Sep 07 22:33:57.570661 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.570579    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:33:57.886779 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:57.886743    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f858c696c-n9mdv"]
Sep 07 22:33:57.893078 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:33:57.893046    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb39ca7_73b6_4fe8_b5b0_7f2dc796a4bf.slice/crio-008aa0936aaf47de922c935f53fc12016b0deed8844d280767641603e084ae96 WatchSource:0}: Error finding container 008aa0936aaf47de922c935f53fc12016b0deed8844d280767641603e084ae96: Status 404 returned error can't find the container with id 008aa0936aaf47de922c935f53fc12016b0deed8844d280767641603e084ae96
Sep 07 22:33:58.082157 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.082077    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efc025f8-0b6c-46b4-a44a-a40e35a8035a","Type":"ContainerStarted","Data":"5300361aa6f0fa88fe4db3ec8a125ee0fa16998f60f2679f7705f27579c4073f"}
Sep 07 22:33:58.082622 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.082160    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efc025f8-0b6c-46b4-a44a-a40e35a8035a","Type":"ContainerStarted","Data":"89e3595e1979c79832dd87f2352c6a8c396ff35e0a901183c84b2dcd12c972c1"}
Sep 07 22:33:58.082622 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.082195    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efc025f8-0b6c-46b4-a44a-a40e35a8035a","Type":"ContainerStarted","Data":"09b950e32e374222ad08386a6ddd6f5f4e19522d62ea64f530a1ddeacb00f6c9"}
Sep 07 22:33:58.082622 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.082229    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efc025f8-0b6c-46b4-a44a-a40e35a8035a","Type":"ContainerStarted","Data":"6bc96d77e3404200edb37fb6565788f73c9e5228882c3926a90576304429959a"}
Sep 07 22:33:58.082622 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.082283    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efc025f8-0b6c-46b4-a44a-a40e35a8035a","Type":"ContainerStarted","Data":"f3ddf83abc49bda0f7876cfb9840ba07f66c5b3479baeb2b8bae12d337558509"}
Sep 07 22:33:58.084414 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.084386    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv" event={"ID":"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf","Type":"ContainerStarted","Data":"008aa0936aaf47de922c935f53fc12016b0deed8844d280767641603e084ae96"}
Sep 07 22:33:58.125143 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.125103    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-76cddc7965-qc5k6"]
Sep 07 22:33:58.128652 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.128631    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.130850 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.130824    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 07 22:33:58.131182 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.131147    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 07 22:33:58.131325 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.131295    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 07 22:33:58.131425 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.131405    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 07 22:33:58.131525 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.131509    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 07 22:33:58.131624 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.131570    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-jfznb\""
Sep 07 22:33:58.135634 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.135608    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 07 22:33:58.140797 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.140773    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76cddc7965-qc5k6"]
Sep 07 22:33:58.237554 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.237516    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn44z\" (UniqueName: \"kubernetes.io/projected/1a794ce9-1039-494e-a794-ef7c45159bdf-kube-api-access-gn44z\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.237745 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.237587    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a794ce9-1039-494e-a794-ef7c45159bdf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.237745 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.237618    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1a794ce9-1039-494e-a794-ef7c45159bdf-telemeter-client-tls\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.237745 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.237644    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a794ce9-1039-494e-a794-ef7c45159bdf-metrics-client-ca\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.237745 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.237674    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1a794ce9-1039-494e-a794-ef7c45159bdf-secret-telemeter-client\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.237745 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.237721    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1a794ce9-1039-494e-a794-ef7c45159bdf-federate-client-tls\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.237938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.237771    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a794ce9-1039-494e-a794-ef7c45159bdf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.237938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.237836    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a794ce9-1039-494e-a794-ef7c45159bdf-serving-certs-ca-bundle\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.339368 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.339267    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a794ce9-1039-494e-a794-ef7c45159bdf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.339368 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.339334    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a794ce9-1039-494e-a794-ef7c45159bdf-serving-certs-ca-bundle\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.339763 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.339407    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn44z\" (UniqueName: \"kubernetes.io/projected/1a794ce9-1039-494e-a794-ef7c45159bdf-kube-api-access-gn44z\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.339763 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.339457    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a794ce9-1039-494e-a794-ef7c45159bdf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.339763 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.339487    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1a794ce9-1039-494e-a794-ef7c45159bdf-telemeter-client-tls\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.339763 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.339694    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a794ce9-1039-494e-a794-ef7c45159bdf-metrics-client-ca\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.339763 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.339746    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1a794ce9-1039-494e-a794-ef7c45159bdf-secret-telemeter-client\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.340006 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.339775    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1a794ce9-1039-494e-a794-ef7c45159bdf-federate-client-tls\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.340218 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.340163    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a794ce9-1039-494e-a794-ef7c45159bdf-serving-certs-ca-bundle\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.340360 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.340332    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a794ce9-1039-494e-a794-ef7c45159bdf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.340530 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.340474    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a794ce9-1039-494e-a794-ef7c45159bdf-metrics-client-ca\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.342723 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.342697    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1a794ce9-1039-494e-a794-ef7c45159bdf-telemeter-client-tls\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.342827 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.342768    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1a794ce9-1039-494e-a794-ef7c45159bdf-federate-client-tls\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.342827 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.342774    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1a794ce9-1039-494e-a794-ef7c45159bdf-secret-telemeter-client\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.342827 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.342807    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a794ce9-1039-494e-a794-ef7c45159bdf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.346791 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.346769    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn44z\" (UniqueName: \"kubernetes.io/projected/1a794ce9-1039-494e-a794-ef7c45159bdf-kube-api-access-gn44z\") pod \"telemeter-client-76cddc7965-qc5k6\" (UID: \"1a794ce9-1039-494e-a794-ef7c45159bdf\") " pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.440826 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.440781    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6"
Sep 07 22:33:58.729078 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:58.728884    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76cddc7965-qc5k6"]
Sep 07 22:33:58.867974 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:33:58.867937    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a794ce9_1039_494e_a794_ef7c45159bdf.slice/crio-ca261be59c0b6972820a5c28da4421b4a36a76d6198494a35c10bd2260d835a2 WatchSource:0}: Error finding container ca261be59c0b6972820a5c28da4421b4a36a76d6198494a35c10bd2260d835a2: Status 404 returned error can't find the container with id ca261be59c0b6972820a5c28da4421b4a36a76d6198494a35c10bd2260d835a2
Sep 07 22:33:59.101673 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.101558    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efc025f8-0b6c-46b4-a44a-a40e35a8035a","Type":"ContainerStarted","Data":"5b46b1902a279c1219bf0b824d0b8d0cf7f98616eedcd2bc3dc9029ad43fe7ca"}
Sep 07 22:33:59.105982 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.105946    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6" event={"ID":"1a794ce9-1039-494e-a794-ef7c45159bdf","Type":"ContainerStarted","Data":"ca261be59c0b6972820a5c28da4421b4a36a76d6198494a35c10bd2260d835a2"}
Sep 07 22:33:59.127178 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.126992    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.5162958509999998 podStartE2EDuration="6.126971278s" totalImagesPullingTime="3.589744557s" totalInitContainerRuntime="20.93087ms" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:59.126376686 +0000 UTC m=+122.183209650" watchObservedRunningTime="2026-09-07 22:33:59.126971278 +0000 UTC m=+122.183804240"
Sep 07 22:33:59.208341 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.208212    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 22:33:59.213622 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.213592    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.216956 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.216645    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 07 22:33:59.217360 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.217233    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 07 22:33:59.217718 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.217637    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 07 22:33:59.221386 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.218219    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4pkukh1u3a3hn\""
Sep 07 22:33:59.221386 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.218229    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 07 22:33:59.221386 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.218502    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 07 22:33:59.221386 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.218674    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 07 22:33:59.221386 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.218688    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 07 22:33:59.221386 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.218898    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 07 22:33:59.221386 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.219074    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 07 22:33:59.221386 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.219125    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 07 22:33:59.221386 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.219264    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 07 22:33:59.224454 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.223591    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-78ck5\""
Sep 07 22:33:59.225269 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.224879    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 07 22:33:59.225979 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.225798    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 07 22:33:59.231997 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.231582    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 07 22:33:59.233712 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.233617    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 22:33:59.260828 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.259808    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.260828 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.259859    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6198080e-9502-4ef3-bbe4-613d8b295581-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.260828 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.259886    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-web-config\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.260828 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.259913    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.260828 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.259943    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.260828 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.259997    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.260828 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.260021    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.260828 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.260045    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwv8v\" (UniqueName: \"kubernetes.io/projected/6198080e-9502-4ef3-bbe4-613d8b295581-kube-api-access-wwv8v\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.260828 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.260087    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.260828 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.260112    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-config\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.260828 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.260143    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.260828 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.260179    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.260828 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.260215    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.260828 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.260299    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.260828 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.260323    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.260828 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.260350    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6198080e-9502-4ef3-bbe4-613d8b295581-config-out\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.261803 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.260379    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.261803 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.260424    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.261803 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.260452    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.261803 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.260485    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.361449 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.360907    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.361449 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.360950    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6198080e-9502-4ef3-bbe4-613d8b295581-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.361449 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.360975    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-web-config\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.361449 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.361119    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.361449 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.361179    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.361449 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.361238    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.361449 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.361285    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.361449 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.361311    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwv8v\" (UniqueName: \"kubernetes.io/projected/6198080e-9502-4ef3-bbe4-613d8b295581-kube-api-access-wwv8v\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.361449 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.361360    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.361449 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.361389    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-config\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.361449 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.361424    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.361449 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.361465    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.362758 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.361493    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.362758 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.361550    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.362758 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.361575    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.362758 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.361604    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6198080e-9502-4ef3-bbe4-613d8b295581-config-out\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.362758 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.361636    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.362758 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.361685    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.362758 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.361711    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.362758 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.361732    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.362758 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.361748    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.362758 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.362149    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.363292 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.362978    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.364043 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.363685    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.365608 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.365228    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.365608 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.365350    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.365608 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.365452    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-config\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.365829 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.365670    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.367466 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.366687    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.367466 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.366799    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.367466 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.367313    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.368270 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.368232    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.368871 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.368840    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.370463 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.370209    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6198080e-9502-4ef3-bbe4-613d8b295581-config-out\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.370844 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.370680    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.370844 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.370802    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6198080e-9502-4ef3-bbe4-613d8b295581-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.371721 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.371334    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.371721 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.371685    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-web-config\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.372855 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.372811    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.373117 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.373093    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwv8v\" (UniqueName: \"kubernetes.io/projected/6198080e-9502-4ef3-bbe4-613d8b295581-kube-api-access-wwv8v\") pod \"prometheus-k8s-0\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.498675 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.498633    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79998dbc84-kskwc"]
Sep 07 22:33:59.537644 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.537481    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:33:59.968304 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:33:59.968114    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 22:33:59.972507 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:33:59.972472    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6198080e_9502_4ef3_bbe4_613d8b295581.slice/crio-ec7f0d4fb74e5de78ca713a22c5e75ba317b44b0cec6deec3e1b0a3cce178c10 WatchSource:0}: Error finding container ec7f0d4fb74e5de78ca713a22c5e75ba317b44b0cec6deec3e1b0a3cce178c10: Status 404 returned error can't find the container with id ec7f0d4fb74e5de78ca713a22c5e75ba317b44b0cec6deec3e1b0a3cce178c10
Sep 07 22:34:00.113776 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:00.113735    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv" event={"ID":"bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf","Type":"ContainerStarted","Data":"6c00703de4331e24e02e7774b47bf5ccfaebd7367ebfa3e3997d7eb5cd4e65aa"}
Sep 07 22:34:00.117089 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:00.117047    2567 generic.go:356] "Generic (PLEG): container finished" podID="6198080e-9502-4ef3-bbe4-613d8b295581" containerID="40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17" exitCode=0
Sep 07 22:34:00.118999 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:00.118970    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6198080e-9502-4ef3-bbe4-613d8b295581","Type":"ContainerDied","Data":"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17"}
Sep 07 22:34:00.119141 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:00.119044    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6198080e-9502-4ef3-bbe4-613d8b295581","Type":"ContainerStarted","Data":"ec7f0d4fb74e5de78ca713a22c5e75ba317b44b0cec6deec3e1b0a3cce178c10"}
Sep 07 22:34:00.132300 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:00.132214    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv" podStartSLOduration=1.2821433309999999 podStartE2EDuration="3.132191738s" totalImagesPullingTime="1.850048407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:34:00.131686749 +0000 UTC m=+123.188519711" watchObservedRunningTime="2026-09-07 22:34:00.132191738 +0000 UTC m=+123.189024701"
Sep 07 22:34:02.130520 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:02.130476    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6" event={"ID":"1a794ce9-1039-494e-a794-ef7c45159bdf","Type":"ContainerStarted","Data":"b98e301d381ef4ff31dc3d97c1282fb5bcf7a670ebbbb98dbc2d031d087f2e66"}
Sep 07 22:34:02.130935 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:02.130551    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6" event={"ID":"1a794ce9-1039-494e-a794-ef7c45159bdf","Type":"ContainerStarted","Data":"95a7a64be802a7b2bf9d9bb3708ffd505adb65d1f84a20c18623598e849d8e17"}
Sep 07 22:34:02.130935 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:02.130572    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6" event={"ID":"1a794ce9-1039-494e-a794-ef7c45159bdf","Type":"ContainerStarted","Data":"a9714a7df1a7d4d440f8137ae2f629caa9e69fc7e08ae405018652cdd7913e90"}
Sep 07 22:34:02.153084 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:02.153024    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-76cddc7965-qc5k6" podStartSLOduration=1.8404678319999999 podStartE2EDuration="4.153000795s" totalImagesPullingTime="2.312532963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:34:02.150536666 +0000 UTC m=+125.207369652" watchObservedRunningTime="2026-09-07 22:34:02.153000795 +0000 UTC m=+125.209833753"
Sep 07 22:34:05.151029 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:05.150995    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6198080e-9502-4ef3-bbe4-613d8b295581","Type":"ContainerStarted","Data":"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d"}
Sep 07 22:34:05.151476 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:05.151078    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6198080e-9502-4ef3-bbe4-613d8b295581","Type":"ContainerStarted","Data":"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2"}
Sep 07 22:34:07.246092 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:07.246045    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs\") pod \"network-metrics-daemon-wrtvx\" (UID: \"f95816cb-c07f-4e9f-b828-930dcf1c06eb\") " pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:34:07.248581 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:07.248550    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f95816cb-c07f-4e9f-b828-930dcf1c06eb-metrics-certs\") pod \"network-metrics-daemon-wrtvx\" (UID: \"f95816cb-c07f-4e9f-b828-930dcf1c06eb\") " pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:34:07.488557 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:07.488519    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lg5n9\""
Sep 07 22:34:07.497298 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:07.497200    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wrtvx"
Sep 07 22:34:14.010353 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:14.010316    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wrtvx"]
Sep 07 22:34:14.016485 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:34:14.016457    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95816cb_c07f_4e9f_b828_930dcf1c06eb.slice/crio-8730b570a3fc8bfad708541d8c1f96ef1d6841f5a6c83209a00100baf03a0396 WatchSource:0}: Error finding container 8730b570a3fc8bfad708541d8c1f96ef1d6841f5a6c83209a00100baf03a0396: Status 404 returned error can't find the container with id 8730b570a3fc8bfad708541d8c1f96ef1d6841f5a6c83209a00100baf03a0396
Sep 07 22:34:14.197647 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:14.197605    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6198080e-9502-4ef3-bbe4-613d8b295581","Type":"ContainerStarted","Data":"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420"}
Sep 07 22:34:14.197846 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:14.197694    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6198080e-9502-4ef3-bbe4-613d8b295581","Type":"ContainerStarted","Data":"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca"}
Sep 07 22:34:14.197846 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:14.197737    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6198080e-9502-4ef3-bbe4-613d8b295581","Type":"ContainerStarted","Data":"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7"}
Sep 07 22:34:14.197846 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:14.197780    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6198080e-9502-4ef3-bbe4-613d8b295581","Type":"ContainerStarted","Data":"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b"}
Sep 07 22:34:14.199693 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:14.199650    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-d6b77" event={"ID":"87bdbb8b-6a3b-4a92-a68e-3ead40f9d615","Type":"ContainerStarted","Data":"11f58afb17c76d0473fd79119b91dc440ac48ac4e9c144bdf4b17bb93c48f2af"}
Sep 07 22:34:14.199817 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:14.199716    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-d6b77"
Sep 07 22:34:14.201308 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:14.201273    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wrtvx" event={"ID":"f95816cb-c07f-4e9f-b828-930dcf1c06eb","Type":"ContainerStarted","Data":"8730b570a3fc8bfad708541d8c1f96ef1d6841f5a6c83209a00100baf03a0396"}
Sep 07 22:34:14.219026 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:14.218992    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-d6b77"
Sep 07 22:34:14.227255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:14.226842    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.75392989 podStartE2EDuration="15.226814796s" totalImagesPullingTime="13.443712449s" totalInitContainerRuntime="29.172457ms" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:34:14.223904247 +0000 UTC m=+137.280737243" watchObservedRunningTime="2026-09-07 22:34:14.226814796 +0000 UTC m=+137.283647765"
Sep 07 22:34:14.241131 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:14.239966    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-d6b77" podStartSLOduration=1.674061373 podStartE2EDuration="20.239942582s" totalImagesPullingTime="18.565881209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:34:14.238696548 +0000 UTC m=+137.295529510" watchObservedRunningTime="2026-09-07 22:34:14.239942582 +0000 UTC m=+137.296775544"
Sep 07 22:34:14.538045 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:14.537883    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:34:15.208295 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:15.208232    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wrtvx" event={"ID":"f95816cb-c07f-4e9f-b828-930dcf1c06eb","Type":"ContainerStarted","Data":"71f08e9efc3b559f0099343221dab87687d64d5e6a4ea7fa4b330258a42e5999"}
Sep 07 22:34:16.215958 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:16.215902    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wrtvx" event={"ID":"f95816cb-c07f-4e9f-b828-930dcf1c06eb","Type":"ContainerStarted","Data":"41d67f5c2c45600a21d9333fc426c5733135c7986d29966fbe804c29ef649fca"}
Sep 07 22:34:16.229929 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:16.229869    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wrtvx" podStartSLOduration=138.210883604 podStartE2EDuration="2m19.229846032s" totalImagesPullingTime="1.018962428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:34:16.229596226 +0000 UTC m=+139.286429188" watchObservedRunningTime="2026-09-07 22:34:16.229846032 +0000 UTC m=+139.286678994"
Sep 07 22:34:17.571762 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:17.571716    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:34:17.572208 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:17.571804    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:34:24.528028 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:24.527985    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79998dbc84-kskwc" podUID="6836bd00-5ba9-4802-b393-0d830fb1eba1" containerName="registry" containerID="cri-o://4cbef3bc2163ec3d526a610dc7e87f60a56760c1844032e9f0edbf1a60f64bc7" gracePeriod=30
Sep 07 22:34:24.814597 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:24.814571    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:34:24.919492 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:24.919451    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls\") pod \"6836bd00-5ba9-4802-b393-0d830fb1eba1\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") "
Sep 07 22:34:24.919672 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:24.919510    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6836bd00-5ba9-4802-b393-0d830fb1eba1-installation-pull-secrets\") pod \"6836bd00-5ba9-4802-b393-0d830fb1eba1\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") "
Sep 07 22:34:24.919672 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:24.919529    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6836bd00-5ba9-4802-b393-0d830fb1eba1-image-registry-private-configuration\") pod \"6836bd00-5ba9-4802-b393-0d830fb1eba1\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") "
Sep 07 22:34:24.919672 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:24.919558    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-bound-sa-token\") pod \"6836bd00-5ba9-4802-b393-0d830fb1eba1\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") "
Sep 07 22:34:24.919672 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:24.919579    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjftc\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-kube-api-access-fjftc\") pod \"6836bd00-5ba9-4802-b393-0d830fb1eba1\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") "
Sep 07 22:34:24.919672 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:24.919605    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-certificates\") pod \"6836bd00-5ba9-4802-b393-0d830fb1eba1\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") "
Sep 07 22:34:24.919941 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:24.919696    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6836bd00-5ba9-4802-b393-0d830fb1eba1-trusted-ca\") pod \"6836bd00-5ba9-4802-b393-0d830fb1eba1\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") "
Sep 07 22:34:24.919941 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:24.919726    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6836bd00-5ba9-4802-b393-0d830fb1eba1-ca-trust-extracted\") pod \"6836bd00-5ba9-4802-b393-0d830fb1eba1\" (UID: \"6836bd00-5ba9-4802-b393-0d830fb1eba1\") "
Sep 07 22:34:24.920155 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:24.920085    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6836bd00-5ba9-4802-b393-0d830fb1eba1" (UID: "6836bd00-5ba9-4802-b393-0d830fb1eba1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:34:24.920532 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:24.920485    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6836bd00-5ba9-4802-b393-0d830fb1eba1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6836bd00-5ba9-4802-b393-0d830fb1eba1" (UID: "6836bd00-5ba9-4802-b393-0d830fb1eba1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:34:24.922386 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:24.922308    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-kube-api-access-fjftc" (OuterVolumeSpecName: "kube-api-access-fjftc") pod "6836bd00-5ba9-4802-b393-0d830fb1eba1" (UID: "6836bd00-5ba9-4802-b393-0d830fb1eba1"). InnerVolumeSpecName "kube-api-access-fjftc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:34:24.922386 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:24.922338    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6836bd00-5ba9-4802-b393-0d830fb1eba1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6836bd00-5ba9-4802-b393-0d830fb1eba1" (UID: "6836bd00-5ba9-4802-b393-0d830fb1eba1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:34:24.922386 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:24.922377    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6836bd00-5ba9-4802-b393-0d830fb1eba1" (UID: "6836bd00-5ba9-4802-b393-0d830fb1eba1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:34:24.922603 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:24.922402    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6836bd00-5ba9-4802-b393-0d830fb1eba1" (UID: "6836bd00-5ba9-4802-b393-0d830fb1eba1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:34:24.922603 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:24.922506    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6836bd00-5ba9-4802-b393-0d830fb1eba1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6836bd00-5ba9-4802-b393-0d830fb1eba1" (UID: "6836bd00-5ba9-4802-b393-0d830fb1eba1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:34:24.929037 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:24.929007    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6836bd00-5ba9-4802-b393-0d830fb1eba1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6836bd00-5ba9-4802-b393-0d830fb1eba1" (UID: "6836bd00-5ba9-4802-b393-0d830fb1eba1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:34:25.021035 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:25.020990    2567 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-bound-sa-token\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:34:25.021035 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:25.021030    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjftc\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-kube-api-access-fjftc\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:34:25.021035 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:25.021043    2567 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-certificates\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:34:25.021338 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:25.021055    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6836bd00-5ba9-4802-b393-0d830fb1eba1-trusted-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:34:25.021338 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:25.021066    2567 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6836bd00-5ba9-4802-b393-0d830fb1eba1-ca-trust-extracted\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:34:25.021338 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:25.021077    2567 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6836bd00-5ba9-4802-b393-0d830fb1eba1-registry-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:34:25.021338 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:25.021087    2567 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6836bd00-5ba9-4802-b393-0d830fb1eba1-installation-pull-secrets\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:34:25.021338 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:25.021098    2567 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6836bd00-5ba9-4802-b393-0d830fb1eba1-image-registry-private-configuration\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:34:25.256709 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:25.256614    2567 generic.go:356] "Generic (PLEG): container finished" podID="6836bd00-5ba9-4802-b393-0d830fb1eba1" containerID="4cbef3bc2163ec3d526a610dc7e87f60a56760c1844032e9f0edbf1a60f64bc7" exitCode=0
Sep 07 22:34:25.256709 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:25.256664    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79998dbc84-kskwc"
Sep 07 22:34:25.256904 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:25.256696    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79998dbc84-kskwc" event={"ID":"6836bd00-5ba9-4802-b393-0d830fb1eba1","Type":"ContainerDied","Data":"4cbef3bc2163ec3d526a610dc7e87f60a56760c1844032e9f0edbf1a60f64bc7"}
Sep 07 22:34:25.256904 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:25.256764    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79998dbc84-kskwc" event={"ID":"6836bd00-5ba9-4802-b393-0d830fb1eba1","Type":"ContainerDied","Data":"e683c3c2736f9b26fb2a401661cf21920136962e5a5618a16cb266430b306a74"}
Sep 07 22:34:25.256904 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:25.256787    2567 scope.go:117] "RemoveContainer" containerID="4cbef3bc2163ec3d526a610dc7e87f60a56760c1844032e9f0edbf1a60f64bc7"
Sep 07 22:34:25.265847 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:25.265828    2567 scope.go:117] "RemoveContainer" containerID="4cbef3bc2163ec3d526a610dc7e87f60a56760c1844032e9f0edbf1a60f64bc7"
Sep 07 22:34:25.266211 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:34:25.266192    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cbef3bc2163ec3d526a610dc7e87f60a56760c1844032e9f0edbf1a60f64bc7\": container with ID starting with 4cbef3bc2163ec3d526a610dc7e87f60a56760c1844032e9f0edbf1a60f64bc7 not found: ID does not exist" containerID="4cbef3bc2163ec3d526a610dc7e87f60a56760c1844032e9f0edbf1a60f64bc7"
Sep 07 22:34:25.266279 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:25.266223    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cbef3bc2163ec3d526a610dc7e87f60a56760c1844032e9f0edbf1a60f64bc7"} err="failed to get container status \"4cbef3bc2163ec3d526a610dc7e87f60a56760c1844032e9f0edbf1a60f64bc7\": rpc error: code = NotFound desc = could not find container \"4cbef3bc2163ec3d526a610dc7e87f60a56760c1844032e9f0edbf1a60f64bc7\": container with ID starting with 4cbef3bc2163ec3d526a610dc7e87f60a56760c1844032e9f0edbf1a60f64bc7 not found: ID does not exist"
Sep 07 22:34:25.277939 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:25.277908    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79998dbc84-kskwc"]
Sep 07 22:34:25.283088 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:25.283061    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79998dbc84-kskwc"]
Sep 07 22:34:25.464008 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:25.463978    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6836bd00-5ba9-4802-b393-0d830fb1eba1" path="/var/lib/kubelet/pods/6836bd00-5ba9-4802-b393-0d830fb1eba1/volumes"
Sep 07 22:34:37.576301 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:37.576236    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:34:37.580228 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:37.580204    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5f858c696c-n9mdv"
Sep 07 22:34:59.538449 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:59.538406    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:34:59.602932 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:34:59.602902    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:00.417518 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:00.417491    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:13.267258 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:13.267205    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 22:35:13.267740 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:13.267521    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="alertmanager" containerID="cri-o://f3ddf83abc49bda0f7876cfb9840ba07f66c5b3479baeb2b8bae12d337558509" gracePeriod=120
Sep 07 22:35:13.267740 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:13.267588    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="kube-rbac-proxy-web" containerID="cri-o://09b950e32e374222ad08386a6ddd6f5f4e19522d62ea64f530a1ddeacb00f6c9" gracePeriod=120
Sep 07 22:35:13.267740 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:13.267634    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="config-reloader" containerID="cri-o://6bc96d77e3404200edb37fb6565788f73c9e5228882c3926a90576304429959a" gracePeriod=120
Sep 07 22:35:13.267740 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:13.267663    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="prom-label-proxy" containerID="cri-o://5b46b1902a279c1219bf0b824d0b8d0cf7f98616eedcd2bc3dc9029ad43fe7ca" gracePeriod=120
Sep 07 22:35:13.267740 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:13.267682    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="kube-rbac-proxy" containerID="cri-o://89e3595e1979c79832dd87f2352c6a8c396ff35e0a901183c84b2dcd12c972c1" gracePeriod=120
Sep 07 22:35:13.267998 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:13.267585    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="kube-rbac-proxy-metric" containerID="cri-o://5300361aa6f0fa88fe4db3ec8a125ee0fa16998f60f2679f7705f27579c4073f" gracePeriod=120
Sep 07 22:35:13.457801 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:13.457764    2567 generic.go:356] "Generic (PLEG): container finished" podID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerID="5b46b1902a279c1219bf0b824d0b8d0cf7f98616eedcd2bc3dc9029ad43fe7ca" exitCode=0
Sep 07 22:35:13.457801 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:13.457789    2567 generic.go:356] "Generic (PLEG): container finished" podID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerID="89e3595e1979c79832dd87f2352c6a8c396ff35e0a901183c84b2dcd12c972c1" exitCode=0
Sep 07 22:35:13.457801 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:13.457795    2567 generic.go:356] "Generic (PLEG): container finished" podID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerID="6bc96d77e3404200edb37fb6565788f73c9e5228882c3926a90576304429959a" exitCode=0
Sep 07 22:35:13.457801 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:13.457802    2567 generic.go:356] "Generic (PLEG): container finished" podID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerID="f3ddf83abc49bda0f7876cfb9840ba07f66c5b3479baeb2b8bae12d337558509" exitCode=0
Sep 07 22:35:13.458045 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:13.457832    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efc025f8-0b6c-46b4-a44a-a40e35a8035a","Type":"ContainerDied","Data":"5b46b1902a279c1219bf0b824d0b8d0cf7f98616eedcd2bc3dc9029ad43fe7ca"}
Sep 07 22:35:13.458045 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:13.457901    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efc025f8-0b6c-46b4-a44a-a40e35a8035a","Type":"ContainerDied","Data":"89e3595e1979c79832dd87f2352c6a8c396ff35e0a901183c84b2dcd12c972c1"}
Sep 07 22:35:13.458045 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:13.457927    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efc025f8-0b6c-46b4-a44a-a40e35a8035a","Type":"ContainerDied","Data":"6bc96d77e3404200edb37fb6565788f73c9e5228882c3926a90576304429959a"}
Sep 07 22:35:13.458045 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:13.457952    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efc025f8-0b6c-46b4-a44a-a40e35a8035a","Type":"ContainerDied","Data":"f3ddf83abc49bda0f7876cfb9840ba07f66c5b3479baeb2b8bae12d337558509"}
Sep 07 22:35:14.481548 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.481266    2567 generic.go:356] "Generic (PLEG): container finished" podID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerID="5300361aa6f0fa88fe4db3ec8a125ee0fa16998f60f2679f7705f27579c4073f" exitCode=0
Sep 07 22:35:14.481548 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.481297    2567 generic.go:356] "Generic (PLEG): container finished" podID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerID="09b950e32e374222ad08386a6ddd6f5f4e19522d62ea64f530a1ddeacb00f6c9" exitCode=0
Sep 07 22:35:14.481548 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.481412    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efc025f8-0b6c-46b4-a44a-a40e35a8035a","Type":"ContainerDied","Data":"5300361aa6f0fa88fe4db3ec8a125ee0fa16998f60f2679f7705f27579c4073f"}
Sep 07 22:35:14.481548 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.481488    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efc025f8-0b6c-46b4-a44a-a40e35a8035a","Type":"ContainerDied","Data":"09b950e32e374222ad08386a6ddd6f5f4e19522d62ea64f530a1ddeacb00f6c9"}
Sep 07 22:35:14.536482 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.536452    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:14.571956 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.571913    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efc025f8-0b6c-46b4-a44a-a40e35a8035a-metrics-client-ca\") pod \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") "
Sep 07 22:35:14.572139 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.571976    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-kube-rbac-proxy\") pod \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") "
Sep 07 22:35:14.572139 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.572026    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") "
Sep 07 22:35:14.572139 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.572064    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efc025f8-0b6c-46b4-a44a-a40e35a8035a-config-out\") pod \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") "
Sep 07 22:35:14.572337 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.572210    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-config-volume\") pod \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") "
Sep 07 22:35:14.572337 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.572300    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-web-config\") pod \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") "
Sep 07 22:35:14.572337 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.572331    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq7dc\" (UniqueName: \"kubernetes.io/projected/efc025f8-0b6c-46b4-a44a-a40e35a8035a-kube-api-access-tq7dc\") pod \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") "
Sep 07 22:35:14.572488 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.572357    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc025f8-0b6c-46b4-a44a-a40e35a8035a-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "efc025f8-0b6c-46b4-a44a-a40e35a8035a" (UID: "efc025f8-0b6c-46b4-a44a-a40e35a8035a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:35:14.572488 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.572369    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-kube-rbac-proxy-web\") pod \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") "
Sep 07 22:35:14.572488 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.572423    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efc025f8-0b6c-46b4-a44a-a40e35a8035a-tls-assets\") pod \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") "
Sep 07 22:35:14.572488 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.572459    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-main-tls\") pod \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") "
Sep 07 22:35:14.572693 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.572488    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/efc025f8-0b6c-46b4-a44a-a40e35a8035a-alertmanager-main-db\") pod \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") "
Sep 07 22:35:14.572693 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.572525    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-cluster-tls-config\") pod \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") "
Sep 07 22:35:14.572693 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.572567    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc025f8-0b6c-46b4-a44a-a40e35a8035a-alertmanager-trusted-ca-bundle\") pod \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\" (UID: \"efc025f8-0b6c-46b4-a44a-a40e35a8035a\") "
Sep 07 22:35:14.573269 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.572870    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efc025f8-0b6c-46b4-a44a-a40e35a8035a-metrics-client-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:14.573591 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.573428    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc025f8-0b6c-46b4-a44a-a40e35a8035a-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "efc025f8-0b6c-46b4-a44a-a40e35a8035a" (UID: "efc025f8-0b6c-46b4-a44a-a40e35a8035a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:35:14.574401 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.574356    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc025f8-0b6c-46b4-a44a-a40e35a8035a-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "efc025f8-0b6c-46b4-a44a-a40e35a8035a" (UID: "efc025f8-0b6c-46b4-a44a-a40e35a8035a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:35:14.576341 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.576309    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "efc025f8-0b6c-46b4-a44a-a40e35a8035a" (UID: "efc025f8-0b6c-46b4-a44a-a40e35a8035a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:35:14.577859 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.577808    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "efc025f8-0b6c-46b4-a44a-a40e35a8035a" (UID: "efc025f8-0b6c-46b4-a44a-a40e35a8035a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:35:14.578304 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.578211    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc025f8-0b6c-46b4-a44a-a40e35a8035a-config-out" (OuterVolumeSpecName: "config-out") pod "efc025f8-0b6c-46b4-a44a-a40e35a8035a" (UID: "efc025f8-0b6c-46b4-a44a-a40e35a8035a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:35:14.578517 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.578476    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "efc025f8-0b6c-46b4-a44a-a40e35a8035a" (UID: "efc025f8-0b6c-46b4-a44a-a40e35a8035a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:35:14.578612 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.578570    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-config-volume" (OuterVolumeSpecName: "config-volume") pod "efc025f8-0b6c-46b4-a44a-a40e35a8035a" (UID: "efc025f8-0b6c-46b4-a44a-a40e35a8035a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:35:14.578673 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.578640    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "efc025f8-0b6c-46b4-a44a-a40e35a8035a" (UID: "efc025f8-0b6c-46b4-a44a-a40e35a8035a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:35:14.578759 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.578738    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc025f8-0b6c-46b4-a44a-a40e35a8035a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "efc025f8-0b6c-46b4-a44a-a40e35a8035a" (UID: "efc025f8-0b6c-46b4-a44a-a40e35a8035a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:35:14.578932 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.578905    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc025f8-0b6c-46b4-a44a-a40e35a8035a-kube-api-access-tq7dc" (OuterVolumeSpecName: "kube-api-access-tq7dc") pod "efc025f8-0b6c-46b4-a44a-a40e35a8035a" (UID: "efc025f8-0b6c-46b4-a44a-a40e35a8035a"). InnerVolumeSpecName "kube-api-access-tq7dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:35:14.583466 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.583439    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "efc025f8-0b6c-46b4-a44a-a40e35a8035a" (UID: "efc025f8-0b6c-46b4-a44a-a40e35a8035a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:35:14.589712 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.589655    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-web-config" (OuterVolumeSpecName: "web-config") pod "efc025f8-0b6c-46b4-a44a-a40e35a8035a" (UID: "efc025f8-0b6c-46b4-a44a-a40e35a8035a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:35:14.673269 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.673186    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:14.673269 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.673236    2567 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efc025f8-0b6c-46b4-a44a-a40e35a8035a-config-out\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:14.673269 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.673276    2567 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:14.673269 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.673285    2567 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-web-config\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:14.673532 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.673296    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq7dc\" (UniqueName: \"kubernetes.io/projected/efc025f8-0b6c-46b4-a44a-a40e35a8035a-kube-api-access-tq7dc\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:14.673532 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.673306    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:14.673532 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.673316    2567 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efc025f8-0b6c-46b4-a44a-a40e35a8035a-tls-assets\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:14.673532 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.673324    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-main-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:14.673532 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.673332    2567 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/efc025f8-0b6c-46b4-a44a-a40e35a8035a-alertmanager-main-db\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:14.673532 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.673340    2567 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-cluster-tls-config\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:14.673532 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.673348    2567 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc025f8-0b6c-46b4-a44a-a40e35a8035a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:14.673532 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:14.673356    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efc025f8-0b6c-46b4-a44a-a40e35a8035a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:15.487968 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.487931    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efc025f8-0b6c-46b4-a44a-a40e35a8035a","Type":"ContainerDied","Data":"66b62eeef12acb854d757225321b3827a93ee0791ec42bb210662e2a83d0c7b4"}
Sep 07 22:35:15.487968 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.487954    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.488445 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.488016    2567 scope.go:117] "RemoveContainer" containerID="5b46b1902a279c1219bf0b824d0b8d0cf7f98616eedcd2bc3dc9029ad43fe7ca"
Sep 07 22:35:15.495920 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.495899    2567 scope.go:117] "RemoveContainer" containerID="5300361aa6f0fa88fe4db3ec8a125ee0fa16998f60f2679f7705f27579c4073f"
Sep 07 22:35:15.503209 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.503189    2567 scope.go:117] "RemoveContainer" containerID="89e3595e1979c79832dd87f2352c6a8c396ff35e0a901183c84b2dcd12c972c1"
Sep 07 22:35:15.509418 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.509386    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 22:35:15.511853 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.511829    2567 scope.go:117] "RemoveContainer" containerID="09b950e32e374222ad08386a6ddd6f5f4e19522d62ea64f530a1ddeacb00f6c9"
Sep 07 22:35:15.512681 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.512660    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 22:35:15.519105 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.519085    2567 scope.go:117] "RemoveContainer" containerID="6bc96d77e3404200edb37fb6565788f73c9e5228882c3926a90576304429959a"
Sep 07 22:35:15.526414 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.526393    2567 scope.go:117] "RemoveContainer" containerID="f3ddf83abc49bda0f7876cfb9840ba07f66c5b3479baeb2b8bae12d337558509"
Sep 07 22:35:15.534205 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.534180    2567 scope.go:117] "RemoveContainer" containerID="ef565aa0a6949fce93e2da8650371f6f25f41550696185376a14a46256cfb94c"
Sep 07 22:35:15.540333 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540304    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 22:35:15.540599 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540578    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="kube-rbac-proxy-metric"
Sep 07 22:35:15.540660 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540602    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="kube-rbac-proxy-metric"
Sep 07 22:35:15.540660 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540615    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="alertmanager"
Sep 07 22:35:15.540660 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540621    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="alertmanager"
Sep 07 22:35:15.540660 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540631    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="kube-rbac-proxy-web"
Sep 07 22:35:15.540660 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540636    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="kube-rbac-proxy-web"
Sep 07 22:35:15.540660 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540642    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="kube-rbac-proxy"
Sep 07 22:35:15.540660 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540649    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="kube-rbac-proxy"
Sep 07 22:35:15.540660 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540662    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="prom-label-proxy"
Sep 07 22:35:15.540908 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540668    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="prom-label-proxy"
Sep 07 22:35:15.540908 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540674    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="config-reloader"
Sep 07 22:35:15.540908 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540679    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="config-reloader"
Sep 07 22:35:15.540908 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540686    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6836bd00-5ba9-4802-b393-0d830fb1eba1" containerName="registry"
Sep 07 22:35:15.540908 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540691    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6836bd00-5ba9-4802-b393-0d830fb1eba1" containerName="registry"
Sep 07 22:35:15.540908 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540701    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="init-config-reloader"
Sep 07 22:35:15.540908 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540706    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="init-config-reloader"
Sep 07 22:35:15.540908 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540747    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="kube-rbac-proxy-web"
Sep 07 22:35:15.540908 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540754    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="kube-rbac-proxy"
Sep 07 22:35:15.540908 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540760    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="config-reloader"
Sep 07 22:35:15.540908 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540767    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6836bd00-5ba9-4802-b393-0d830fb1eba1" containerName="registry"
Sep 07 22:35:15.540908 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540772    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="alertmanager"
Sep 07 22:35:15.540908 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540778    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="kube-rbac-proxy-metric"
Sep 07 22:35:15.540908 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.540784    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" containerName="prom-label-proxy"
Sep 07 22:35:15.545184 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.545164    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.548167 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.548140    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 07 22:35:15.548280 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.548155    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 07 22:35:15.548602 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.548571    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 07 22:35:15.548672 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.548606    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 07 22:35:15.548734 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.548607    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 07 22:35:15.548734 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.548575    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 07 22:35:15.548816 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.548614    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qllvm\""
Sep 07 22:35:15.548960 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.548944    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 07 22:35:15.550062 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.550021    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 07 22:35:15.555703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.555682    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 07 22:35:15.572907 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.572871    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 22:35:15.581881 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.581841    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-config-volume\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.582060 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.581954    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.582060 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.582017    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.582060 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.582054    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.582224 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.582078    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.582224 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.582139    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-web-config\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.582224 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.582171    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.582224 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.582189    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.582224 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.582216    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbf6c\" (UniqueName: \"kubernetes.io/projected/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-kube-api-access-fbf6c\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.582456 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.582296    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.582456 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.582336    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-config-out\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.582525 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.582449    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-tls-assets\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.582525 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.582490    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.683677 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.683580    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-tls-assets\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.683677 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.683620    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.683677 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.683639    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-config-volume\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.683677 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.683668    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.683988 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.683696    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.683988 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.683723    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.683988 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.683743    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.683988 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.683781    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-web-config\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.683988 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.683804    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.683988 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.683830    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.683988 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.683852    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbf6c\" (UniqueName: \"kubernetes.io/projected/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-kube-api-access-fbf6c\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.683988 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.683875    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.683988 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.683909    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-config-out\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.686171 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.684738    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.686171 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.685299    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.686171 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.686113    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.686863 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.686808    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-tls-assets\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.686983 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.686873    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-config-volume\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.687079 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.687052    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.687221 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.687198    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.687464 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.687444    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-config-out\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.687717 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.687692    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.688197 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.688170    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.688494 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.688473    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-web-config\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.688778 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.688760    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.693824 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.693802    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbf6c\" (UniqueName: \"kubernetes.io/projected/daff1dfd-5a2e-4aab-a65d-70d20d7edf49-kube-api-access-fbf6c\") pod \"alertmanager-main-0\" (UID: \"daff1dfd-5a2e-4aab-a65d-70d20d7edf49\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:15.855480 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:15.855434    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 22:35:16.136171 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:16.136129    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 22:35:16.139547 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:35:16.139521    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaff1dfd_5a2e_4aab_a65d_70d20d7edf49.slice/crio-2e2ea5c0722e4dce8feee7b9c900cdddaeae493e2573f4e7a409f882c18d4329 WatchSource:0}: Error finding container 2e2ea5c0722e4dce8feee7b9c900cdddaeae493e2573f4e7a409f882c18d4329: Status 404 returned error can't find the container with id 2e2ea5c0722e4dce8feee7b9c900cdddaeae493e2573f4e7a409f882c18d4329
Sep 07 22:35:16.493935 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:16.493842    2567 generic.go:356] "Generic (PLEG): container finished" podID="daff1dfd-5a2e-4aab-a65d-70d20d7edf49" containerID="3d9068ae3b2db59de2c3ab36240fa0ccdc686a7823a16984e3a867322d85afb0" exitCode=0
Sep 07 22:35:16.493935 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:16.493879    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"daff1dfd-5a2e-4aab-a65d-70d20d7edf49","Type":"ContainerDied","Data":"3d9068ae3b2db59de2c3ab36240fa0ccdc686a7823a16984e3a867322d85afb0"}
Sep 07 22:35:16.494406 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:16.493947    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"daff1dfd-5a2e-4aab-a65d-70d20d7edf49","Type":"ContainerStarted","Data":"2e2ea5c0722e4dce8feee7b9c900cdddaeae493e2573f4e7a409f882c18d4329"}
Sep 07 22:35:17.464835 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.464798    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efc025f8-0b6c-46b4-a44a-a40e35a8035a" path="/var/lib/kubelet/pods/efc025f8-0b6c-46b4-a44a-a40e35a8035a/volumes"
Sep 07 22:35:17.499748 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.499713    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 22:35:17.500195 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.500078    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="prometheus" containerID="cri-o://656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2" gracePeriod=600
Sep 07 22:35:17.500195 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.500117    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420" gracePeriod=600
Sep 07 22:35:17.500195 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.500183    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="thanos-sidecar" containerID="cri-o://d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b" gracePeriod=600
Sep 07 22:35:17.500434 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.500228    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="config-reloader" containerID="cri-o://f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d" gracePeriod=600
Sep 07 22:35:17.500434 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.500295    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="kube-rbac-proxy" containerID="cri-o://6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca" gracePeriod=600
Sep 07 22:35:17.500434 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.500369    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="kube-rbac-proxy-web" containerID="cri-o://3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7" gracePeriod=600
Sep 07 22:35:17.517125 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.516407    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"daff1dfd-5a2e-4aab-a65d-70d20d7edf49","Type":"ContainerStarted","Data":"4517c8f2c31107a46cb9c177207313d965085666eb5a99dc90d9a5ca67497be2"}
Sep 07 22:35:17.517125 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.516501    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"daff1dfd-5a2e-4aab-a65d-70d20d7edf49","Type":"ContainerStarted","Data":"d3fe4d72aae456ff3e369dfd0b96403a3a7d2c6166e223ee39039f7adc5eeefd"}
Sep 07 22:35:17.517125 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.516538    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"daff1dfd-5a2e-4aab-a65d-70d20d7edf49","Type":"ContainerStarted","Data":"4440a5449712874ae6939a207a05f0b2b02789638b23614aa96ef89c55a47fc4"}
Sep 07 22:35:17.517125 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.516575    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"daff1dfd-5a2e-4aab-a65d-70d20d7edf49","Type":"ContainerStarted","Data":"e97bc0b056f8505cc1056b38a19e340b80e28b2930d31116608406b4fcb1ac77"}
Sep 07 22:35:17.517125 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.516618    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"daff1dfd-5a2e-4aab-a65d-70d20d7edf49","Type":"ContainerStarted","Data":"aea69548136f38b1dd41677940b2ae8c9dfbbaef9c80b9c05814145352408ea1"}
Sep 07 22:35:17.517125 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.516653    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"daff1dfd-5a2e-4aab-a65d-70d20d7edf49","Type":"ContainerStarted","Data":"16d0496ebf2f08503ab9240e89804d618096b57515f7c7010937f31bad9d3134"}
Sep 07 22:35:17.569586 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.569517    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.553515795 podStartE2EDuration="2.569491266s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.975471ms" isStatefulPod=false podCreationTimestamp="2026-09-07 22:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:35:17.567637463 +0000 UTC m=+200.624470458" watchObservedRunningTime="2026-09-07 22:35:17.569491266 +0000 UTC m=+200.626324230"
Sep 07 22:35:17.760756 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.760715    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:17.805207 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.805167    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6198080e-9502-4ef3-bbe4-613d8b295581-config-out\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.805420 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.805223    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-metrics-client-certs\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.805582 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.805558    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6198080e-9502-4ef3-bbe4-613d8b295581-tls-assets\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.805672 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.805602    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-rulefiles-1\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.805672 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.805631    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-config\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.805672 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.805660    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-prometheus-k8s-tls\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.805823 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.805692    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-db\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.805823 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.805717    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-kube-rbac-proxy\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.805823 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.805753    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.805823 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.805812    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-trusted-ca-bundle\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.806019 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.805859    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-configmap-metrics-client-ca\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.806019 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.805885    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwv8v\" (UniqueName: \"kubernetes.io/projected/6198080e-9502-4ef3-bbe4-613d8b295581-kube-api-access-wwv8v\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.806019 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.805911    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-configmap-kubelet-serving-ca-bundle\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.806019 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.805960    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.806019 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.806008    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-grpc-tls\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.806260 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.806040    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-thanos-prometheus-http-client-file\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.806260 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.806079    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-web-config\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.806260 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.806106    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-rulefiles-0\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.806260 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.806143    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-rulefiles-2\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.806260 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.806169    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-configmap-serving-certs-ca-bundle\") pod \"6198080e-9502-4ef3-bbe4-613d8b295581\" (UID: \"6198080e-9502-4ef3-bbe4-613d8b295581\") "
Sep 07 22:35:17.807739 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.806759    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:35:17.807739 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.806844    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:35:17.808467 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.808286    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:35:17.809220 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.808852    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:35:17.809334 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.809291    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:35:17.809865 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.809837    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:35:17.809946 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.809916    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6198080e-9502-4ef3-bbe4-613d8b295581-config-out" (OuterVolumeSpecName: "config-out") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:35:17.810538 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.810452    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:35:17.810722 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.810685    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:35:17.812339 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.812302    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6198080e-9502-4ef3-bbe4-613d8b295581-kube-api-access-wwv8v" (OuterVolumeSpecName: "kube-api-access-wwv8v") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "kube-api-access-wwv8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:35:17.813220 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.813182    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:35:17.813467 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.813427    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-config" (OuterVolumeSpecName: "config") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:35:17.813572 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.813549    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:35:17.813872 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.813714    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:35:17.813872 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.813791    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:35:17.813872 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.813870    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6198080e-9502-4ef3-bbe4-613d8b295581-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:35:17.814218 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.814181    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:35:17.815140 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.814958    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:35:17.815659 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.815631    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:35:17.824031 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.823992    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-web-config" (OuterVolumeSpecName: "web-config") pod "6198080e-9502-4ef3-bbe4-613d8b295581" (UID: "6198080e-9502-4ef3-bbe4-613d8b295581"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:35:17.907699 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907649    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:17.907699 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907690    2567 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-grpc-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:17.907699 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907702    2567 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:17.907931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907714    2567 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-web-config\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:17.907931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907725    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:17.907931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907733    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:17.907931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907742    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:17.907931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907750    2567 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6198080e-9502-4ef3-bbe4-613d8b295581-config-out\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:17.907931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907758    2567 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-metrics-client-certs\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:17.907931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907766    2567 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6198080e-9502-4ef3-bbe4-613d8b295581-tls-assets\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:17.907931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907775    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:17.907931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907783    2567 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-config\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:17.907931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907792    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:17.907931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907800    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-k8s-db\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:17.907931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907808    2567 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-kube-rbac-proxy\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:17.907931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907816    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6198080e-9502-4ef3-bbe4-613d8b295581-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:17.907931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907824    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:17.907931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907832    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-configmap-metrics-client-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:17.907931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907840    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwv8v\" (UniqueName: \"kubernetes.io/projected/6198080e-9502-4ef3-bbe4-613d8b295581-kube-api-access-wwv8v\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:17.907931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:17.907848    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6198080e-9502-4ef3-bbe4-613d8b295581-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:35:18.523712 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.523672    2567 generic.go:356] "Generic (PLEG): container finished" podID="6198080e-9502-4ef3-bbe4-613d8b295581" containerID="2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420" exitCode=0
Sep 07 22:35:18.523712 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.523704    2567 generic.go:356] "Generic (PLEG): container finished" podID="6198080e-9502-4ef3-bbe4-613d8b295581" containerID="6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca" exitCode=0
Sep 07 22:35:18.523712 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.523712    2567 generic.go:356] "Generic (PLEG): container finished" podID="6198080e-9502-4ef3-bbe4-613d8b295581" containerID="3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7" exitCode=0
Sep 07 22:35:18.524206 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.523723    2567 generic.go:356] "Generic (PLEG): container finished" podID="6198080e-9502-4ef3-bbe4-613d8b295581" containerID="d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b" exitCode=0
Sep 07 22:35:18.524206 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.523731    2567 generic.go:356] "Generic (PLEG): container finished" podID="6198080e-9502-4ef3-bbe4-613d8b295581" containerID="f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d" exitCode=0
Sep 07 22:35:18.524206 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.523739    2567 generic.go:356] "Generic (PLEG): container finished" podID="6198080e-9502-4ef3-bbe4-613d8b295581" containerID="656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2" exitCode=0
Sep 07 22:35:18.524206 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.523754    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6198080e-9502-4ef3-bbe4-613d8b295581","Type":"ContainerDied","Data":"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420"}
Sep 07 22:35:18.524206 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.523824    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6198080e-9502-4ef3-bbe4-613d8b295581","Type":"ContainerDied","Data":"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca"}
Sep 07 22:35:18.524206 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.523766    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.524206 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.523851    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6198080e-9502-4ef3-bbe4-613d8b295581","Type":"ContainerDied","Data":"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7"}
Sep 07 22:35:18.524206 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.523878    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6198080e-9502-4ef3-bbe4-613d8b295581","Type":"ContainerDied","Data":"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b"}
Sep 07 22:35:18.524206 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.523921    2567 scope.go:117] "RemoveContainer" containerID="2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420"
Sep 07 22:35:18.524206 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.523930    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6198080e-9502-4ef3-bbe4-613d8b295581","Type":"ContainerDied","Data":"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d"}
Sep 07 22:35:18.524206 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.523955    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6198080e-9502-4ef3-bbe4-613d8b295581","Type":"ContainerDied","Data":"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2"}
Sep 07 22:35:18.524206 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.523979    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6198080e-9502-4ef3-bbe4-613d8b295581","Type":"ContainerDied","Data":"ec7f0d4fb74e5de78ca713a22c5e75ba317b44b0cec6deec3e1b0a3cce178c10"}
Sep 07 22:35:18.532462 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.532309    2567 scope.go:117] "RemoveContainer" containerID="6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca"
Sep 07 22:35:18.540653 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.540631    2567 scope.go:117] "RemoveContainer" containerID="3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7"
Sep 07 22:35:18.547561 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.547536    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 22:35:18.548683 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.548666    2567 scope.go:117] "RemoveContainer" containerID="d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b"
Sep 07 22:35:18.552018 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.551994    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 22:35:18.556670 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.556647    2567 scope.go:117] "RemoveContainer" containerID="f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d"
Sep 07 22:35:18.563856 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.563829    2567 scope.go:117] "RemoveContainer" containerID="656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2"
Sep 07 22:35:18.571616 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.571591    2567 scope.go:117] "RemoveContainer" containerID="40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17"
Sep 07 22:35:18.575555 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.575534    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 22:35:18.575847 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.575834    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="kube-rbac-proxy-thanos"
Sep 07 22:35:18.575896 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.575849    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="kube-rbac-proxy-thanos"
Sep 07 22:35:18.575896 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.575860    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="kube-rbac-proxy"
Sep 07 22:35:18.575896 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.575866    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="kube-rbac-proxy"
Sep 07 22:35:18.575896 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.575876    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="init-config-reloader"
Sep 07 22:35:18.575896 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.575885    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="init-config-reloader"
Sep 07 22:35:18.575896 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.575892    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="kube-rbac-proxy-web"
Sep 07 22:35:18.575896 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.575897    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="kube-rbac-proxy-web"
Sep 07 22:35:18.576116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.575908    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="prometheus"
Sep 07 22:35:18.576116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.575913    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="prometheus"
Sep 07 22:35:18.576116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.575926    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="config-reloader"
Sep 07 22:35:18.576116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.575935    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="config-reloader"
Sep 07 22:35:18.576116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.575944    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="thanos-sidecar"
Sep 07 22:35:18.576116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.575951    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="thanos-sidecar"
Sep 07 22:35:18.576116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.575999    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="kube-rbac-proxy"
Sep 07 22:35:18.576116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.576006    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="config-reloader"
Sep 07 22:35:18.576116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.576012    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="kube-rbac-proxy-thanos"
Sep 07 22:35:18.576116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.576018    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="thanos-sidecar"
Sep 07 22:35:18.576116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.576024    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="kube-rbac-proxy-web"
Sep 07 22:35:18.576116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.576029    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" containerName="prometheus"
Sep 07 22:35:18.579004 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.578982    2567 scope.go:117] "RemoveContainer" containerID="2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420"
Sep 07 22:35:18.579301 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:35:18.579272    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420\": container with ID starting with 2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420 not found: ID does not exist" containerID="2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420"
Sep 07 22:35:18.579362 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.579303    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420"} err="failed to get container status \"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420\": rpc error: code = NotFound desc = could not find container \"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420\": container with ID starting with 2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420 not found: ID does not exist"
Sep 07 22:35:18.579362 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.579320    2567 scope.go:117] "RemoveContainer" containerID="6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca"
Sep 07 22:35:18.579599 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:35:18.579582    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca\": container with ID starting with 6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca not found: ID does not exist" containerID="6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca"
Sep 07 22:35:18.579649 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.579604    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca"} err="failed to get container status \"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca\": rpc error: code = NotFound desc = could not find container \"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca\": container with ID starting with 6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca not found: ID does not exist"
Sep 07 22:35:18.579649 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.579620    2567 scope.go:117] "RemoveContainer" containerID="3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7"
Sep 07 22:35:18.579869 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:35:18.579852    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7\": container with ID starting with 3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7 not found: ID does not exist" containerID="3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7"
Sep 07 22:35:18.579926 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.579878    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7"} err="failed to get container status \"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7\": rpc error: code = NotFound desc = could not find container \"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7\": container with ID starting with 3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7 not found: ID does not exist"
Sep 07 22:35:18.579926 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.579899    2567 scope.go:117] "RemoveContainer" containerID="d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b"
Sep 07 22:35:18.580144 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:35:18.580126    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b\": container with ID starting with d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b not found: ID does not exist" containerID="d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b"
Sep 07 22:35:18.580197 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.580148    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b"} err="failed to get container status \"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b\": rpc error: code = NotFound desc = could not find container \"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b\": container with ID starting with d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b not found: ID does not exist"
Sep 07 22:35:18.580197 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.580161    2567 scope.go:117] "RemoveContainer" containerID="f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d"
Sep 07 22:35:18.580408 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:35:18.580394    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d\": container with ID starting with f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d not found: ID does not exist" containerID="f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d"
Sep 07 22:35:18.580451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.580416    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d"} err="failed to get container status \"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d\": rpc error: code = NotFound desc = could not find container \"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d\": container with ID starting with f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d not found: ID does not exist"
Sep 07 22:35:18.580451 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.580438    2567 scope.go:117] "RemoveContainer" containerID="656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2"
Sep 07 22:35:18.580737 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:35:18.580718    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2\": container with ID starting with 656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2 not found: ID does not exist" containerID="656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2"
Sep 07 22:35:18.580830 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.580745    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2"} err="failed to get container status \"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2\": rpc error: code = NotFound desc = could not find container \"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2\": container with ID starting with 656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2 not found: ID does not exist"
Sep 07 22:35:18.580888 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.580841    2567 scope.go:117] "RemoveContainer" containerID="40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17"
Sep 07 22:35:18.580978 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.580956    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.581394 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:35:18.581181    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17\": container with ID starting with 40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17 not found: ID does not exist" containerID="40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17"
Sep 07 22:35:18.581394 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.581211    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17"} err="failed to get container status \"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17\": rpc error: code = NotFound desc = could not find container \"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17\": container with ID starting with 40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17 not found: ID does not exist"
Sep 07 22:35:18.581394 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.581233    2567 scope.go:117] "RemoveContainer" containerID="2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420"
Sep 07 22:35:18.581625 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.581599    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420"} err="failed to get container status \"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420\": rpc error: code = NotFound desc = could not find container \"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420\": container with ID starting with 2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420 not found: ID does not exist"
Sep 07 22:35:18.581625 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.581625    2567 scope.go:117] "RemoveContainer" containerID="6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca"
Sep 07 22:35:18.581909 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.581877    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca"} err="failed to get container status \"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca\": rpc error: code = NotFound desc = could not find container \"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca\": container with ID starting with 6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca not found: ID does not exist"
Sep 07 22:35:18.581909 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.581907    2567 scope.go:117] "RemoveContainer" containerID="3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7"
Sep 07 22:35:18.582295 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.582272    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7"} err="failed to get container status \"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7\": rpc error: code = NotFound desc = could not find container \"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7\": container with ID starting with 3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7 not found: ID does not exist"
Sep 07 22:35:18.582295 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.582295    2567 scope.go:117] "RemoveContainer" containerID="d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b"
Sep 07 22:35:18.582783 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.582761    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b"} err="failed to get container status \"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b\": rpc error: code = NotFound desc = could not find container \"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b\": container with ID starting with d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b not found: ID does not exist"
Sep 07 22:35:18.582783 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.582782    2567 scope.go:117] "RemoveContainer" containerID="f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d"
Sep 07 22:35:18.583134 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.583076    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d"} err="failed to get container status \"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d\": rpc error: code = NotFound desc = could not find container \"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d\": container with ID starting with f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d not found: ID does not exist"
Sep 07 22:35:18.583134 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.583102    2567 scope.go:117] "RemoveContainer" containerID="656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2"
Sep 07 22:35:18.583427 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.583401    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2"} err="failed to get container status \"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2\": rpc error: code = NotFound desc = could not find container \"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2\": container with ID starting with 656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2 not found: ID does not exist"
Sep 07 22:35:18.583490 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.583429    2567 scope.go:117] "RemoveContainer" containerID="40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17"
Sep 07 22:35:18.583490 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.583440    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 07 22:35:18.583490 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.583414    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 07 22:35:18.583621 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.583570    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-78ck5\""
Sep 07 22:35:18.583621 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.583609    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 07 22:35:18.583795 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.583774    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17"} err="failed to get container status \"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17\": rpc error: code = NotFound desc = could not find container \"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17\": container with ID starting with 40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17 not found: ID does not exist"
Sep 07 22:35:18.583842 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.583797    2567 scope.go:117] "RemoveContainer" containerID="2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420"
Sep 07 22:35:18.583956 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.583889    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 07 22:35:18.583956 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.583925    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 07 22:35:18.584528 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.584133    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 07 22:35:18.584528 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.584141    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4pkukh1u3a3hn\""
Sep 07 22:35:18.584528 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.584189    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 07 22:35:18.584528 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.584206    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 07 22:35:18.584528 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.584140    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 07 22:35:18.584528 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.584313    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 07 22:35:18.584528 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.584384    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 07 22:35:18.584528 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.584305    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420"} err="failed to get container status \"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420\": rpc error: code = NotFound desc = could not find container \"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420\": container with ID starting with 2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420 not found: ID does not exist"
Sep 07 22:35:18.584528 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.584422    2567 scope.go:117] "RemoveContainer" containerID="6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca"
Sep 07 22:35:18.584528 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.584408    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 07 22:35:18.585171 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.584725    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca"} err="failed to get container status \"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca\": rpc error: code = NotFound desc = could not find container \"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca\": container with ID starting with 6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca not found: ID does not exist"
Sep 07 22:35:18.585171 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.584752    2567 scope.go:117] "RemoveContainer" containerID="3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7"
Sep 07 22:35:18.585171 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.585077    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7"} err="failed to get container status \"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7\": rpc error: code = NotFound desc = could not find container \"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7\": container with ID starting with 3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7 not found: ID does not exist"
Sep 07 22:35:18.585171 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.585102    2567 scope.go:117] "RemoveContainer" containerID="d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b"
Sep 07 22:35:18.585502 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.585460    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b"} err="failed to get container status \"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b\": rpc error: code = NotFound desc = could not find container \"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b\": container with ID starting with d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b not found: ID does not exist"
Sep 07 22:35:18.585502 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.585488    2567 scope.go:117] "RemoveContainer" containerID="f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d"
Sep 07 22:35:18.585828 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.585808    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d"} err="failed to get container status \"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d\": rpc error: code = NotFound desc = could not find container \"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d\": container with ID starting with f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d not found: ID does not exist"
Sep 07 22:35:18.585828 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.585827    2567 scope.go:117] "RemoveContainer" containerID="656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2"
Sep 07 22:35:18.586291 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.586166    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2"} err="failed to get container status \"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2\": rpc error: code = NotFound desc = could not find container \"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2\": container with ID starting with 656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2 not found: ID does not exist"
Sep 07 22:35:18.586291 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.586193    2567 scope.go:117] "RemoveContainer" containerID="40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17"
Sep 07 22:35:18.587154 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.586514    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 07 22:35:18.587154 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.586571    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17"} err="failed to get container status \"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17\": rpc error: code = NotFound desc = could not find container \"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17\": container with ID starting with 40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17 not found: ID does not exist"
Sep 07 22:35:18.587154 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.586590    2567 scope.go:117] "RemoveContainer" containerID="2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420"
Sep 07 22:35:18.587154 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.586994    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420"} err="failed to get container status \"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420\": rpc error: code = NotFound desc = could not find container \"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420\": container with ID starting with 2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420 not found: ID does not exist"
Sep 07 22:35:18.587154 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.587020    2567 scope.go:117] "RemoveContainer" containerID="6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca"
Sep 07 22:35:18.588124 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.587768    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca"} err="failed to get container status \"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca\": rpc error: code = NotFound desc = could not find container \"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca\": container with ID starting with 6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca not found: ID does not exist"
Sep 07 22:35:18.588124 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.587792    2567 scope.go:117] "RemoveContainer" containerID="3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7"
Sep 07 22:35:18.588678 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.588383    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7"} err="failed to get container status \"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7\": rpc error: code = NotFound desc = could not find container \"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7\": container with ID starting with 3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7 not found: ID does not exist"
Sep 07 22:35:18.588678 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.588406    2567 scope.go:117] "RemoveContainer" containerID="d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b"
Sep 07 22:35:18.589607 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.589413    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b"} err="failed to get container status \"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b\": rpc error: code = NotFound desc = could not find container \"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b\": container with ID starting with d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b not found: ID does not exist"
Sep 07 22:35:18.589607 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.589441    2567 scope.go:117] "RemoveContainer" containerID="f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d"
Sep 07 22:35:18.589846 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.589725    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d"} err="failed to get container status \"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d\": rpc error: code = NotFound desc = could not find container \"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d\": container with ID starting with f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d not found: ID does not exist"
Sep 07 22:35:18.589846 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.589749    2567 scope.go:117] "RemoveContainer" containerID="656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2"
Sep 07 22:35:18.590833 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.590019    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2"} err="failed to get container status \"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2\": rpc error: code = NotFound desc = could not find container \"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2\": container with ID starting with 656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2 not found: ID does not exist"
Sep 07 22:35:18.590833 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.590040    2567 scope.go:117] "RemoveContainer" containerID="40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17"
Sep 07 22:35:18.590833 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.590513    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17"} err="failed to get container status \"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17\": rpc error: code = NotFound desc = could not find container \"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17\": container with ID starting with 40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17 not found: ID does not exist"
Sep 07 22:35:18.590833 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.590542    2567 scope.go:117] "RemoveContainer" containerID="2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420"
Sep 07 22:35:18.591069 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.591043    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420"} err="failed to get container status \"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420\": rpc error: code = NotFound desc = could not find container \"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420\": container with ID starting with 2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420 not found: ID does not exist"
Sep 07 22:35:18.591113 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.591073    2567 scope.go:117] "RemoveContainer" containerID="6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca"
Sep 07 22:35:18.591113 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.591059    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 07 22:35:18.592112 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.592085    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca"} err="failed to get container status \"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca\": rpc error: code = NotFound desc = could not find container \"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca\": container with ID starting with 6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca not found: ID does not exist"
Sep 07 22:35:18.592112 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.592113    2567 scope.go:117] "RemoveContainer" containerID="3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7"
Sep 07 22:35:18.592589 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.592473    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7"} err="failed to get container status \"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7\": rpc error: code = NotFound desc = could not find container \"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7\": container with ID starting with 3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7 not found: ID does not exist"
Sep 07 22:35:18.592589 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.592501    2567 scope.go:117] "RemoveContainer" containerID="d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b"
Sep 07 22:35:18.592840 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.592815    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b"} err="failed to get container status \"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b\": rpc error: code = NotFound desc = could not find container \"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b\": container with ID starting with d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b not found: ID does not exist"
Sep 07 22:35:18.592913 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.592839    2567 scope.go:117] "RemoveContainer" containerID="f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d"
Sep 07 22:35:18.593103 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.593083    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d"} err="failed to get container status \"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d\": rpc error: code = NotFound desc = could not find container \"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d\": container with ID starting with f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d not found: ID does not exist"
Sep 07 22:35:18.593155 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.593105    2567 scope.go:117] "RemoveContainer" containerID="656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2"
Sep 07 22:35:18.593328 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.593313    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 22:35:18.593454 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.593429    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2"} err="failed to get container status \"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2\": rpc error: code = NotFound desc = could not find container \"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2\": container with ID starting with 656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2 not found: ID does not exist"
Sep 07 22:35:18.593454 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.593450    2567 scope.go:117] "RemoveContainer" containerID="40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17"
Sep 07 22:35:18.593699 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.593681    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17"} err="failed to get container status \"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17\": rpc error: code = NotFound desc = could not find container \"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17\": container with ID starting with 40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17 not found: ID does not exist"
Sep 07 22:35:18.593775 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.593698    2567 scope.go:117] "RemoveContainer" containerID="2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420"
Sep 07 22:35:18.593965 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.593943    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420"} err="failed to get container status \"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420\": rpc error: code = NotFound desc = could not find container \"2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420\": container with ID starting with 2967b12fec318b2b90f1a0040db3cd7eb204b21f3de91fd45bb6950055d86420 not found: ID does not exist"
Sep 07 22:35:18.594040 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.593965    2567 scope.go:117] "RemoveContainer" containerID="6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca"
Sep 07 22:35:18.594265 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.594222    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca"} err="failed to get container status \"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca\": rpc error: code = NotFound desc = could not find container \"6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca\": container with ID starting with 6f2fb5f237d183454ef337c32cca80cf4e4d49bc44810708cda328c31152b7ca not found: ID does not exist"
Sep 07 22:35:18.594347 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.594265    2567 scope.go:117] "RemoveContainer" containerID="3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7"
Sep 07 22:35:18.594521 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.594491    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7"} err="failed to get container status \"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7\": rpc error: code = NotFound desc = could not find container \"3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7\": container with ID starting with 3dd7a5146fb9a3639c034255fadc17be14c5fb7f01a4aace4570e42df20082f7 not found: ID does not exist"
Sep 07 22:35:18.594564 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.594523    2567 scope.go:117] "RemoveContainer" containerID="d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b"
Sep 07 22:35:18.594738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.594721    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b"} err="failed to get container status \"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b\": rpc error: code = NotFound desc = could not find container \"d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b\": container with ID starting with d677388a68322b29ab2174d69f8b7bdba912949f9db248a71df6e50fc40c3e8b not found: ID does not exist"
Sep 07 22:35:18.594790 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.594738    2567 scope.go:117] "RemoveContainer" containerID="f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d"
Sep 07 22:35:18.594958 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.594937    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d"} err="failed to get container status \"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d\": rpc error: code = NotFound desc = could not find container \"f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d\": container with ID starting with f76284506bf25bc025103b0da4cfe1f8da03b68d2632ddd98bfc43ee57fb406d not found: ID does not exist"
Sep 07 22:35:18.595055 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.594958    2567 scope.go:117] "RemoveContainer" containerID="656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2"
Sep 07 22:35:18.595281 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.595236    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2"} err="failed to get container status \"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2\": rpc error: code = NotFound desc = could not find container \"656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2\": container with ID starting with 656cfc7501e6531d348afc62c0aef7a564b4eb6cf70c7238f739627b8bf1fce2 not found: ID does not exist"
Sep 07 22:35:18.595281 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.595275    2567 scope.go:117] "RemoveContainer" containerID="40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17"
Sep 07 22:35:18.595556 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.595508    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17"} err="failed to get container status \"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17\": rpc error: code = NotFound desc = could not find container \"40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17\": container with ID starting with 40023fa778b9b11875aa95b6dbbbac5a122357c6111d095167d950b81cf17f17 not found: ID does not exist"
Sep 07 22:35:18.612336 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.612294    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.612532 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.612338    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/91025335-55b9-45b6-ad36-079659f69853-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.612532 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.612423    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.612532 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.612488    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.612532 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.612528    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.612710 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.612558    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.612710 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.612591    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.612710 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.612634    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.612710 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.612681    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.612710 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.612707    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.612884 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.612765    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91025335-55b9-45b6-ad36-079659f69853-config-out\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.612884 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.612787    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.612884 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.612834    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.613029 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.612957    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.613029 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.612997    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbjrr\" (UniqueName: \"kubernetes.io/projected/91025335-55b9-45b6-ad36-079659f69853-kube-api-access-gbjrr\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.613140 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.613058    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-web-config\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.613140 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.613092    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-config\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.613140 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.613134    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91025335-55b9-45b6-ad36-079659f69853-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.613312 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.613153    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.613312 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.613231    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.714286 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714218    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.714286 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714286    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.714650 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714316    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.714650 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714333    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.714650 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714349    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.714650 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714368    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.714650 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714383    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.714650 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714501    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91025335-55b9-45b6-ad36-079659f69853-config-out\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.714650 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714541    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.714650 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714559    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.714650 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714593    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.714650 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714633    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbjrr\" (UniqueName: \"kubernetes.io/projected/91025335-55b9-45b6-ad36-079659f69853-kube-api-access-gbjrr\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.715144 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714675    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-web-config\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.715144 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714707    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-config\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.715144 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714739    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91025335-55b9-45b6-ad36-079659f69853-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.715144 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714768    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.715144 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714809    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.715144 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714867    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.715144 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714897    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/91025335-55b9-45b6-ad36-079659f69853-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.715144 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.714927    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.715144 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.715093    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.717798 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.715912    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.717798 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.716272    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.717798 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.716395    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/91025335-55b9-45b6-ad36-079659f69853-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.717798 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.716580    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.717798 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.716951    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.717798 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.717233    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.717798 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.717482    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.717798 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.717737    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.718370 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.718209    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.719223 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.719196    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.719675 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.719629    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-config\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.719788 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.719732    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.719856 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.719784    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.719856 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.719814    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91025335-55b9-45b6-ad36-079659f69853-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.720457 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.720433    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-web-config\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.720523 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.720448    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/91025335-55b9-45b6-ad36-079659f69853-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.721172 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.721152    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91025335-55b9-45b6-ad36-079659f69853-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.721304 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.721286    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91025335-55b9-45b6-ad36-079659f69853-config-out\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.724074 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.724052    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbjrr\" (UniqueName: \"kubernetes.io/projected/91025335-55b9-45b6-ad36-079659f69853-kube-api-access-gbjrr\") pod \"prometheus-k8s-0\" (UID: \"91025335-55b9-45b6-ad36-079659f69853\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:18.893990 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:18.893893    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:35:19.175999 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:19.175970    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 22:35:19.178371 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:35:19.178348    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91025335_55b9_45b6_ad36_079659f69853.slice/crio-d7d130fc1a10f850cc169d07f3ce9f234427941b80c8f7fe2f3b5a7607c679e5 WatchSource:0}: Error finding container d7d130fc1a10f850cc169d07f3ce9f234427941b80c8f7fe2f3b5a7607c679e5: Status 404 returned error can't find the container with id d7d130fc1a10f850cc169d07f3ce9f234427941b80c8f7fe2f3b5a7607c679e5
Sep 07 22:35:19.465096 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:19.465013    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6198080e-9502-4ef3-bbe4-613d8b295581" path="/var/lib/kubelet/pods/6198080e-9502-4ef3-bbe4-613d8b295581/volumes"
Sep 07 22:35:19.530702 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:19.530662    2567 generic.go:356] "Generic (PLEG): container finished" podID="91025335-55b9-45b6-ad36-079659f69853" containerID="102416b54354c92c7b5586745881e5ce70c2a0fb815bf1818e13cada0f874ea3" exitCode=0
Sep 07 22:35:19.531075 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:19.530713    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91025335-55b9-45b6-ad36-079659f69853","Type":"ContainerDied","Data":"102416b54354c92c7b5586745881e5ce70c2a0fb815bf1818e13cada0f874ea3"}
Sep 07 22:35:19.531075 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:19.530769    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91025335-55b9-45b6-ad36-079659f69853","Type":"ContainerStarted","Data":"d7d130fc1a10f850cc169d07f3ce9f234427941b80c8f7fe2f3b5a7607c679e5"}
Sep 07 22:35:20.539476 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:20.539441    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91025335-55b9-45b6-ad36-079659f69853","Type":"ContainerStarted","Data":"8bdfb1570b14a4957100b595fa0c35bc60e5f24a0470943fac12ced00d3af179"}
Sep 07 22:35:20.539873 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:20.539506    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91025335-55b9-45b6-ad36-079659f69853","Type":"ContainerStarted","Data":"cc2fd5c642d7d887aaaf191fc0f2f921741e409951b297634d2f33dfdb649b6d"}
Sep 07 22:35:20.539873 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:20.539532    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91025335-55b9-45b6-ad36-079659f69853","Type":"ContainerStarted","Data":"8b8353677291baf958c19987cc39782a8b5197dc6abbf2e2d7e2054c8b2ec82d"}
Sep 07 22:35:20.539873 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:20.539558    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91025335-55b9-45b6-ad36-079659f69853","Type":"ContainerStarted","Data":"08d7b9f52b1abcb1e14651bbe2d6decc71016bc98dc30affe906ff8c0a9ab933"}
Sep 07 22:35:20.539873 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:20.539580    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91025335-55b9-45b6-ad36-079659f69853","Type":"ContainerStarted","Data":"d00b91806c46b197455d472801401419838c21ed85e04f1190e443f47b417ea4"}
Sep 07 22:35:20.539873 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:20.539602    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91025335-55b9-45b6-ad36-079659f69853","Type":"ContainerStarted","Data":"6a8656fa22daa8ca5a6bea6433918a883e889c7ca4f961e531e78c98c81f0453"}
Sep 07 22:35:20.566201 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:20.566129    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.5449602430000002 podStartE2EDuration="2.566105428s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.145185ms" isStatefulPod=false podCreationTimestamp="2026-09-07 22:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:35:20.564115483 +0000 UTC m=+203.620948438" watchObservedRunningTime="2026-09-07 22:35:20.566105428 +0000 UTC m=+203.622938390"
Sep 07 22:35:23.894968 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:35:23.894917    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:36:18.894415 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:18.894377    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:36:18.910169 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:18.910139    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:36:19.785139 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:19.785106    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 22:36:33.822409 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:33.822365    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xpw6l"]
Sep 07 22:36:33.825439 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:33.825421    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xpw6l"
Sep 07 22:36:33.827693 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:33.827674    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 07 22:36:33.828357 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:33.828329    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 07 22:36:33.828357 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:33.828344    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-wzcfr\""
Sep 07 22:36:33.834123 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:33.834099    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xpw6l"]
Sep 07 22:36:33.957172 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:33.957132    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c3ed7a6-300c-4ac1-8434-671d3478d42b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xpw6l\" (UID: \"9c3ed7a6-300c-4ac1-8434-671d3478d42b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xpw6l"
Sep 07 22:36:33.957172 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:33.957176    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8smv\" (UniqueName: \"kubernetes.io/projected/9c3ed7a6-300c-4ac1-8434-671d3478d42b-kube-api-access-x8smv\") pod \"cert-manager-cainjector-559cdd8c87-xpw6l\" (UID: \"9c3ed7a6-300c-4ac1-8434-671d3478d42b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xpw6l"
Sep 07 22:36:34.057674 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:34.057630    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c3ed7a6-300c-4ac1-8434-671d3478d42b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xpw6l\" (UID: \"9c3ed7a6-300c-4ac1-8434-671d3478d42b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xpw6l"
Sep 07 22:36:34.057856 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:34.057680    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8smv\" (UniqueName: \"kubernetes.io/projected/9c3ed7a6-300c-4ac1-8434-671d3478d42b-kube-api-access-x8smv\") pod \"cert-manager-cainjector-559cdd8c87-xpw6l\" (UID: \"9c3ed7a6-300c-4ac1-8434-671d3478d42b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xpw6l"
Sep 07 22:36:34.064828 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:34.064798    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c3ed7a6-300c-4ac1-8434-671d3478d42b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xpw6l\" (UID: \"9c3ed7a6-300c-4ac1-8434-671d3478d42b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xpw6l"
Sep 07 22:36:34.064975 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:34.064952    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8smv\" (UniqueName: \"kubernetes.io/projected/9c3ed7a6-300c-4ac1-8434-671d3478d42b-kube-api-access-x8smv\") pod \"cert-manager-cainjector-559cdd8c87-xpw6l\" (UID: \"9c3ed7a6-300c-4ac1-8434-671d3478d42b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xpw6l"
Sep 07 22:36:34.134943 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:34.134853    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xpw6l"
Sep 07 22:36:34.405145 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:34.405032    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xpw6l"]
Sep 07 22:36:34.407765 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:36:34.407737    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c3ed7a6_300c_4ac1_8434_671d3478d42b.slice/crio-5e1675c749c69ed734bcb0f712abfca93a54c9849ce92272ffc5b1b426bdf29a WatchSource:0}: Error finding container 5e1675c749c69ed734bcb0f712abfca93a54c9849ce92272ffc5b1b426bdf29a: Status 404 returned error can't find the container with id 5e1675c749c69ed734bcb0f712abfca93a54c9849ce92272ffc5b1b426bdf29a
Sep 07 22:36:34.829421 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:34.829383    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xpw6l" event={"ID":"9c3ed7a6-300c-4ac1-8434-671d3478d42b","Type":"ContainerStarted","Data":"5e1675c749c69ed734bcb0f712abfca93a54c9849ce92272ffc5b1b426bdf29a"}
Sep 07 22:36:37.846058 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:37.846023    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xpw6l" event={"ID":"9c3ed7a6-300c-4ac1-8434-671d3478d42b","Type":"ContainerStarted","Data":"f95072b2809766cdb87a4bc5bfb0441f6b1c3ad515babb1ce507d9b2f2a009e0"}
Sep 07 22:36:37.861318 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:37.861263    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xpw6l" podStartSLOduration=1.873967167 podStartE2EDuration="4.861231623s" totalImagesPullingTime="2.987264456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:36:37.859384383 +0000 UTC m=+280.916217345" watchObservedRunningTime="2026-09-07 22:36:37.861231623 +0000 UTC m=+280.918064583"
Sep 07 22:36:57.366733 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:57.366702    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/2.log"
Sep 07 22:36:57.370142 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:57.370115    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/2.log"
Sep 07 22:36:57.377993 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:36:57.377969    2567 kubelet.go:1648] "Image garbage collection succeeded"
Sep 07 22:37:09.681200 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.681162    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"]
Sep 07 22:37:09.684614 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.684596    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"
Sep 07 22:37:09.686874 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.686848    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 07 22:37:09.687046 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.686939    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 07 22:37:09.687218 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.687201    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 07 22:37:09.687305 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.687287    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 07 22:37:09.687367 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.687330    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wdhlh\""
Sep 07 22:37:09.707483 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.707443    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"]
Sep 07 22:37:09.755674 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.755637    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aee02241-2234-4d9e-a6c5-3ae7b3ad8d13-tmp\") pod \"opendatahub-operator-controller-manager-7d648d744d-s9jnn\" (UID: \"aee02241-2234-4d9e-a6c5-3ae7b3ad8d13\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"
Sep 07 22:37:09.755674 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.755677    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aee02241-2234-4d9e-a6c5-3ae7b3ad8d13-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d648d744d-s9jnn\" (UID: \"aee02241-2234-4d9e-a6c5-3ae7b3ad8d13\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"
Sep 07 22:37:09.755881 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.755705    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aee02241-2234-4d9e-a6c5-3ae7b3ad8d13-manifests\") pod \"opendatahub-operator-controller-manager-7d648d744d-s9jnn\" (UID: \"aee02241-2234-4d9e-a6c5-3ae7b3ad8d13\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"
Sep 07 22:37:09.755881 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.755825    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aee02241-2234-4d9e-a6c5-3ae7b3ad8d13-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d648d744d-s9jnn\" (UID: \"aee02241-2234-4d9e-a6c5-3ae7b3ad8d13\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"
Sep 07 22:37:09.755881 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.755865    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn8mp\" (UniqueName: \"kubernetes.io/projected/aee02241-2234-4d9e-a6c5-3ae7b3ad8d13-kube-api-access-nn8mp\") pod \"opendatahub-operator-controller-manager-7d648d744d-s9jnn\" (UID: \"aee02241-2234-4d9e-a6c5-3ae7b3ad8d13\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"
Sep 07 22:37:09.856972 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.856932    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aee02241-2234-4d9e-a6c5-3ae7b3ad8d13-tmp\") pod \"opendatahub-operator-controller-manager-7d648d744d-s9jnn\" (UID: \"aee02241-2234-4d9e-a6c5-3ae7b3ad8d13\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"
Sep 07 22:37:09.856972 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.856976    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aee02241-2234-4d9e-a6c5-3ae7b3ad8d13-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d648d744d-s9jnn\" (UID: \"aee02241-2234-4d9e-a6c5-3ae7b3ad8d13\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"
Sep 07 22:37:09.857208 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.857016    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aee02241-2234-4d9e-a6c5-3ae7b3ad8d13-manifests\") pod \"opendatahub-operator-controller-manager-7d648d744d-s9jnn\" (UID: \"aee02241-2234-4d9e-a6c5-3ae7b3ad8d13\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"
Sep 07 22:37:09.857208 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.857110    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aee02241-2234-4d9e-a6c5-3ae7b3ad8d13-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d648d744d-s9jnn\" (UID: \"aee02241-2234-4d9e-a6c5-3ae7b3ad8d13\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"
Sep 07 22:37:09.857208 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.857155    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn8mp\" (UniqueName: \"kubernetes.io/projected/aee02241-2234-4d9e-a6c5-3ae7b3ad8d13-kube-api-access-nn8mp\") pod \"opendatahub-operator-controller-manager-7d648d744d-s9jnn\" (UID: \"aee02241-2234-4d9e-a6c5-3ae7b3ad8d13\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"
Sep 07 22:37:09.857548 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.857523    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aee02241-2234-4d9e-a6c5-3ae7b3ad8d13-manifests\") pod \"opendatahub-operator-controller-manager-7d648d744d-s9jnn\" (UID: \"aee02241-2234-4d9e-a6c5-3ae7b3ad8d13\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"
Sep 07 22:37:09.859521 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.859493    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aee02241-2234-4d9e-a6c5-3ae7b3ad8d13-tmp\") pod \"opendatahub-operator-controller-manager-7d648d744d-s9jnn\" (UID: \"aee02241-2234-4d9e-a6c5-3ae7b3ad8d13\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"
Sep 07 22:37:09.859610 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.859540    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aee02241-2234-4d9e-a6c5-3ae7b3ad8d13-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d648d744d-s9jnn\" (UID: \"aee02241-2234-4d9e-a6c5-3ae7b3ad8d13\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"
Sep 07 22:37:09.859651 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.859621    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aee02241-2234-4d9e-a6c5-3ae7b3ad8d13-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d648d744d-s9jnn\" (UID: \"aee02241-2234-4d9e-a6c5-3ae7b3ad8d13\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"
Sep 07 22:37:09.868260 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.868220    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn8mp\" (UniqueName: \"kubernetes.io/projected/aee02241-2234-4d9e-a6c5-3ae7b3ad8d13-kube-api-access-nn8mp\") pod \"opendatahub-operator-controller-manager-7d648d744d-s9jnn\" (UID: \"aee02241-2234-4d9e-a6c5-3ae7b3ad8d13\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"
Sep 07 22:37:09.995350 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:09.995228    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"
Sep 07 22:37:10.275538 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.275443    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"]
Sep 07 22:37:10.278953 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:37:10.278927    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee02241_2234_4d9e_a6c5_3ae7b3ad8d13.slice/crio-4874ff45c7fed39c773b1cba3aaa26bd49cf78531b332b41471fe0661387e242 WatchSource:0}: Error finding container 4874ff45c7fed39c773b1cba3aaa26bd49cf78531b332b41471fe0661387e242: Status 404 returned error can't find the container with id 4874ff45c7fed39c773b1cba3aaa26bd49cf78531b332b41471fe0661387e242
Sep 07 22:37:10.280831 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.280812    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 22:37:10.287932 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.287906    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn"]
Sep 07 22:37:10.292272 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.292225    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn"
Sep 07 22:37:10.294895 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.294870    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 07 22:37:10.295062 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.294883    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9s5f6\""
Sep 07 22:37:10.295062 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.295007    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 07 22:37:10.295266 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.295230    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 07 22:37:10.295371 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.295305    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 07 22:37:10.295371 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.295331    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 07 22:37:10.304196 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.304166    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn"]
Sep 07 22:37:10.361489 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.361449    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/40abc3f9-3165-4fe2-a06b-90483abc6ab2-metrics-cert\") pod \"lws-controller-manager-575c877bb5-zm9xn\" (UID: \"40abc3f9-3165-4fe2-a06b-90483abc6ab2\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn"
Sep 07 22:37:10.361489 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.361496    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxlwc\" (UniqueName: \"kubernetes.io/projected/40abc3f9-3165-4fe2-a06b-90483abc6ab2-kube-api-access-dxlwc\") pod \"lws-controller-manager-575c877bb5-zm9xn\" (UID: \"40abc3f9-3165-4fe2-a06b-90483abc6ab2\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn"
Sep 07 22:37:10.361718 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.361534    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/40abc3f9-3165-4fe2-a06b-90483abc6ab2-manager-config\") pod \"lws-controller-manager-575c877bb5-zm9xn\" (UID: \"40abc3f9-3165-4fe2-a06b-90483abc6ab2\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn"
Sep 07 22:37:10.361718 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.361566    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40abc3f9-3165-4fe2-a06b-90483abc6ab2-cert\") pod \"lws-controller-manager-575c877bb5-zm9xn\" (UID: \"40abc3f9-3165-4fe2-a06b-90483abc6ab2\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn"
Sep 07 22:37:10.462260 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.462216    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40abc3f9-3165-4fe2-a06b-90483abc6ab2-cert\") pod \"lws-controller-manager-575c877bb5-zm9xn\" (UID: \"40abc3f9-3165-4fe2-a06b-90483abc6ab2\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn"
Sep 07 22:37:10.462468 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.462322    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/40abc3f9-3165-4fe2-a06b-90483abc6ab2-metrics-cert\") pod \"lws-controller-manager-575c877bb5-zm9xn\" (UID: \"40abc3f9-3165-4fe2-a06b-90483abc6ab2\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn"
Sep 07 22:37:10.462468 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.462358    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxlwc\" (UniqueName: \"kubernetes.io/projected/40abc3f9-3165-4fe2-a06b-90483abc6ab2-kube-api-access-dxlwc\") pod \"lws-controller-manager-575c877bb5-zm9xn\" (UID: \"40abc3f9-3165-4fe2-a06b-90483abc6ab2\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn"
Sep 07 22:37:10.462468 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.462403    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/40abc3f9-3165-4fe2-a06b-90483abc6ab2-manager-config\") pod \"lws-controller-manager-575c877bb5-zm9xn\" (UID: \"40abc3f9-3165-4fe2-a06b-90483abc6ab2\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn"
Sep 07 22:37:10.463033 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.463008    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/40abc3f9-3165-4fe2-a06b-90483abc6ab2-manager-config\") pod \"lws-controller-manager-575c877bb5-zm9xn\" (UID: \"40abc3f9-3165-4fe2-a06b-90483abc6ab2\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn"
Sep 07 22:37:10.464909 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.464888    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/40abc3f9-3165-4fe2-a06b-90483abc6ab2-metrics-cert\") pod \"lws-controller-manager-575c877bb5-zm9xn\" (UID: \"40abc3f9-3165-4fe2-a06b-90483abc6ab2\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn"
Sep 07 22:37:10.464967 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.464955    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40abc3f9-3165-4fe2-a06b-90483abc6ab2-cert\") pod \"lws-controller-manager-575c877bb5-zm9xn\" (UID: \"40abc3f9-3165-4fe2-a06b-90483abc6ab2\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn"
Sep 07 22:37:10.470664 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.470642    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxlwc\" (UniqueName: \"kubernetes.io/projected/40abc3f9-3165-4fe2-a06b-90483abc6ab2-kube-api-access-dxlwc\") pod \"lws-controller-manager-575c877bb5-zm9xn\" (UID: \"40abc3f9-3165-4fe2-a06b-90483abc6ab2\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn"
Sep 07 22:37:10.603146 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.603090    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn"
Sep 07 22:37:10.890764 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.890478    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn"]
Sep 07 22:37:10.894282 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:37:10.894133    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40abc3f9_3165_4fe2_a06b_90483abc6ab2.slice/crio-631b2f2476ba79a10007df1845b9ebdb034d71abf07e79e73867114e9f3b0cec WatchSource:0}: Error finding container 631b2f2476ba79a10007df1845b9ebdb034d71abf07e79e73867114e9f3b0cec: Status 404 returned error can't find the container with id 631b2f2476ba79a10007df1845b9ebdb034d71abf07e79e73867114e9f3b0cec
Sep 07 22:37:10.981131 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.981002    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn" event={"ID":"aee02241-2234-4d9e-a6c5-3ae7b3ad8d13","Type":"ContainerStarted","Data":"4874ff45c7fed39c773b1cba3aaa26bd49cf78531b332b41471fe0661387e242"}
Sep 07 22:37:10.983705 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:10.983643    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn" event={"ID":"40abc3f9-3165-4fe2-a06b-90483abc6ab2","Type":"ContainerStarted","Data":"631b2f2476ba79a10007df1845b9ebdb034d71abf07e79e73867114e9f3b0cec"}
Sep 07 22:37:14.001289 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:14.001231    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn" event={"ID":"aee02241-2234-4d9e-a6c5-3ae7b3ad8d13","Type":"ContainerStarted","Data":"40b5791d97e275ef6bff0feb0e9a20ecb78d0bb9762345c43ff202e13a72569c"}
Sep 07 22:37:15.007834 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:15.007799    2567 generic.go:356] "Generic (PLEG): container finished" podID="aee02241-2234-4d9e-a6c5-3ae7b3ad8d13" containerID="40b5791d97e275ef6bff0feb0e9a20ecb78d0bb9762345c43ff202e13a72569c" exitCode=0
Sep 07 22:37:15.008166 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:15.007845    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn" event={"ID":"aee02241-2234-4d9e-a6c5-3ae7b3ad8d13","Type":"ContainerDied","Data":"40b5791d97e275ef6bff0feb0e9a20ecb78d0bb9762345c43ff202e13a72569c"}
Sep 07 22:37:16.014784 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:16.014748    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn" event={"ID":"aee02241-2234-4d9e-a6c5-3ae7b3ad8d13","Type":"ContainerStarted","Data":"6d248002f437b0e09c4ffc433e1e1d08174f063579ddf833d1276deda220c152"}
Sep 07 22:37:16.015219 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:16.014841    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"
Sep 07 22:37:16.016368 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:16.016344    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn" event={"ID":"40abc3f9-3165-4fe2-a06b-90483abc6ab2","Type":"ContainerStarted","Data":"b674a9d198d7f736d7380dfe4e7db526d5e0c47c423ffd9b80467271fe025b1a"}
Sep 07 22:37:16.016486 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:16.016473    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn"
Sep 07 22:37:16.035420 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:16.035349    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn" podStartSLOduration=2.399524504 podStartE2EDuration="7.035325054s" totalImagesPullingTime="3.276979175s" totalInitContainerRuntime="1.358821375s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:37:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:37:16.034281427 +0000 UTC m=+319.091114385" watchObservedRunningTime="2026-09-07 22:37:16.035325054 +0000 UTC m=+319.092158016"
Sep 07 22:37:16.049676 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:16.049604    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn" podStartSLOduration=1.97100802 podStartE2EDuration="6.049585112s" totalImagesPullingTime="4.078577092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:37:16.048189931 +0000 UTC m=+319.105022892" watchObservedRunningTime="2026-09-07 22:37:16.049585112 +0000 UTC m=+319.106418071"
Sep 07 22:37:27.022629 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:27.022593    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-575c877bb5-zm9xn"
Sep 07 22:37:27.023051 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:27.022834    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-s9jnn"
Sep 07 22:37:29.976103 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:29.976059    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68c66dfc89-vdgsz"]
Sep 07 22:37:29.987256 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:29.987213    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68c66dfc89-vdgsz"
Sep 07 22:37:29.990174 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:29.990133    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 07 22:37:29.990440 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:29.990399    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-f7cgl\""
Sep 07 22:37:29.990976 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:29.990748    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 07 22:37:29.990976 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:29.990857    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 07 22:37:29.990976 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:29.990864    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 07 22:37:29.991718 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:29.991691    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68c66dfc89-vdgsz"]
Sep 07 22:37:30.036027 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:30.035988    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh8rg\" (UniqueName: \"kubernetes.io/projected/f0f0ca3c-6829-439a-b38b-642d0b090c9a-kube-api-access-rh8rg\") pod \"kube-auth-proxy-68c66dfc89-vdgsz\" (UID: \"f0f0ca3c-6829-439a-b38b-642d0b090c9a\") " pod="openshift-ingress/kube-auth-proxy-68c66dfc89-vdgsz"
Sep 07 22:37:30.036206 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:30.036039    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0f0ca3c-6829-439a-b38b-642d0b090c9a-tls-certs\") pod \"kube-auth-proxy-68c66dfc89-vdgsz\" (UID: \"f0f0ca3c-6829-439a-b38b-642d0b090c9a\") " pod="openshift-ingress/kube-auth-proxy-68c66dfc89-vdgsz"
Sep 07 22:37:30.036206 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:30.036092    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0f0ca3c-6829-439a-b38b-642d0b090c9a-tmp\") pod \"kube-auth-proxy-68c66dfc89-vdgsz\" (UID: \"f0f0ca3c-6829-439a-b38b-642d0b090c9a\") " pod="openshift-ingress/kube-auth-proxy-68c66dfc89-vdgsz"
Sep 07 22:37:30.137574 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:30.137538    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh8rg\" (UniqueName: \"kubernetes.io/projected/f0f0ca3c-6829-439a-b38b-642d0b090c9a-kube-api-access-rh8rg\") pod \"kube-auth-proxy-68c66dfc89-vdgsz\" (UID: \"f0f0ca3c-6829-439a-b38b-642d0b090c9a\") " pod="openshift-ingress/kube-auth-proxy-68c66dfc89-vdgsz"
Sep 07 22:37:30.137790 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:30.137589    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0f0ca3c-6829-439a-b38b-642d0b090c9a-tls-certs\") pod \"kube-auth-proxy-68c66dfc89-vdgsz\" (UID: \"f0f0ca3c-6829-439a-b38b-642d0b090c9a\") " pod="openshift-ingress/kube-auth-proxy-68c66dfc89-vdgsz"
Sep 07 22:37:30.137790 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:30.137672    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0f0ca3c-6829-439a-b38b-642d0b090c9a-tmp\") pod \"kube-auth-proxy-68c66dfc89-vdgsz\" (UID: \"f0f0ca3c-6829-439a-b38b-642d0b090c9a\") " pod="openshift-ingress/kube-auth-proxy-68c66dfc89-vdgsz"
Sep 07 22:37:30.140021 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:30.139992    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0f0ca3c-6829-439a-b38b-642d0b090c9a-tmp\") pod \"kube-auth-proxy-68c66dfc89-vdgsz\" (UID: \"f0f0ca3c-6829-439a-b38b-642d0b090c9a\") " pod="openshift-ingress/kube-auth-proxy-68c66dfc89-vdgsz"
Sep 07 22:37:30.140226 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:30.140207    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0f0ca3c-6829-439a-b38b-642d0b090c9a-tls-certs\") pod \"kube-auth-proxy-68c66dfc89-vdgsz\" (UID: \"f0f0ca3c-6829-439a-b38b-642d0b090c9a\") " pod="openshift-ingress/kube-auth-proxy-68c66dfc89-vdgsz"
Sep 07 22:37:30.145742 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:30.145713    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh8rg\" (UniqueName: \"kubernetes.io/projected/f0f0ca3c-6829-439a-b38b-642d0b090c9a-kube-api-access-rh8rg\") pod \"kube-auth-proxy-68c66dfc89-vdgsz\" (UID: \"f0f0ca3c-6829-439a-b38b-642d0b090c9a\") " pod="openshift-ingress/kube-auth-proxy-68c66dfc89-vdgsz"
Sep 07 22:37:30.298412 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:30.298317    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68c66dfc89-vdgsz"
Sep 07 22:37:30.585062 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:30.585030    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68c66dfc89-vdgsz"]
Sep 07 22:37:30.587782 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:37:30.587750    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f0ca3c_6829_439a_b38b_642d0b090c9a.slice/crio-af0785dd583838815402b9cb7f25b643ccecb3fb7c4250938f61dbe75ec21bf3 WatchSource:0}: Error finding container af0785dd583838815402b9cb7f25b643ccecb3fb7c4250938f61dbe75ec21bf3: Status 404 returned error can't find the container with id af0785dd583838815402b9cb7f25b643ccecb3fb7c4250938f61dbe75ec21bf3
Sep 07 22:37:31.081516 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:31.081478    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68c66dfc89-vdgsz" event={"ID":"f0f0ca3c-6829-439a-b38b-642d0b090c9a","Type":"ContainerStarted","Data":"af0785dd583838815402b9cb7f25b643ccecb3fb7c4250938f61dbe75ec21bf3"}
Sep 07 22:37:34.098891 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:34.098844    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68c66dfc89-vdgsz" event={"ID":"f0f0ca3c-6829-439a-b38b-642d0b090c9a","Type":"ContainerStarted","Data":"3f4793cd7fabf30a8ad5b5c1fb803c896b47420f0edcd2b953262f9d4b9e7c6a"}
Sep 07 22:37:34.113543 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:37:34.113487    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68c66dfc89-vdgsz" podStartSLOduration=2.45177831 podStartE2EDuration="5.113470058s" totalImagesPullingTime="2.661691748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:37:34.112017904 +0000 UTC m=+337.168850869" watchObservedRunningTime="2026-09-07 22:37:34.113470058 +0000 UTC m=+337.170303042"
Sep 07 22:39:18.134117 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.134078    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-brb4b"]
Sep 07 22:39:18.137129 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.137111    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-brb4b"
Sep 07 22:39:18.139827 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.139805    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 07 22:39:18.139961 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.139805    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 07 22:39:18.139961 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.139851    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cczlv\""
Sep 07 22:39:18.151007 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.150979    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-brb4b"]
Sep 07 22:39:18.202933 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.202887    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmhdp\" (UniqueName: \"kubernetes.io/projected/8ae8a497-0cb2-4d45-8433-525235d960c4-kube-api-access-hmhdp\") pod \"kuadrant-operator-controller-manager-6bd956ccf-brb4b\" (UID: \"8ae8a497-0cb2-4d45-8433-525235d960c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-brb4b"
Sep 07 22:39:18.203113 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.202947    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ae8a497-0cb2-4d45-8433-525235d960c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-brb4b\" (UID: \"8ae8a497-0cb2-4d45-8433-525235d960c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-brb4b"
Sep 07 22:39:18.303876 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.303837    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhdp\" (UniqueName: \"kubernetes.io/projected/8ae8a497-0cb2-4d45-8433-525235d960c4-kube-api-access-hmhdp\") pod \"kuadrant-operator-controller-manager-6bd956ccf-brb4b\" (UID: \"8ae8a497-0cb2-4d45-8433-525235d960c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-brb4b"
Sep 07 22:39:18.304056 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.303892    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ae8a497-0cb2-4d45-8433-525235d960c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-brb4b\" (UID: \"8ae8a497-0cb2-4d45-8433-525235d960c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-brb4b"
Sep 07 22:39:18.304378 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.304357    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ae8a497-0cb2-4d45-8433-525235d960c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-brb4b\" (UID: \"8ae8a497-0cb2-4d45-8433-525235d960c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-brb4b"
Sep 07 22:39:18.336070 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.336038    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmhdp\" (UniqueName: \"kubernetes.io/projected/8ae8a497-0cb2-4d45-8433-525235d960c4-kube-api-access-hmhdp\") pod \"kuadrant-operator-controller-manager-6bd956ccf-brb4b\" (UID: \"8ae8a497-0cb2-4d45-8433-525235d960c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-brb4b"
Sep 07 22:39:18.447556 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.447465    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-brb4b"
Sep 07 22:39:18.735236 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.735209    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-brb4b"]
Sep 07 22:39:18.738293 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:39:18.738260    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae8a497_0cb2_4d45_8433_525235d960c4.slice/crio-f371579c0a1c8504467dbe8c4e67a0cf28f122e10dc6e52bf7a982cdde947b19 WatchSource:0}: Error finding container f371579c0a1c8504467dbe8c4e67a0cf28f122e10dc6e52bf7a982cdde947b19: Status 404 returned error can't find the container with id f371579c0a1c8504467dbe8c4e67a0cf28f122e10dc6e52bf7a982cdde947b19
Sep 07 22:39:18.801159 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.801121    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-brb4b"]
Sep 07 22:39:18.808625 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.808586    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-brb4b"]
Sep 07 22:39:18.830998 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.830958    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb"]
Sep 07 22:39:18.836346 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.836308    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb"
Sep 07 22:39:18.850201 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.850166    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb"]
Sep 07 22:39:18.909564 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.909531    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98851745-710d-424b-bd4a-71a03f95d957-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-cbqcb\" (UID: \"98851745-710d-424b-bd4a-71a03f95d957\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb"
Sep 07 22:39:18.909741 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:18.909575    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht2kd\" (UniqueName: \"kubernetes.io/projected/98851745-710d-424b-bd4a-71a03f95d957-kube-api-access-ht2kd\") pod \"kuadrant-operator-controller-manager-6bd956ccf-cbqcb\" (UID: \"98851745-710d-424b-bd4a-71a03f95d957\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb"
Sep 07 22:39:19.010740 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:19.010642    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98851745-710d-424b-bd4a-71a03f95d957-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-cbqcb\" (UID: \"98851745-710d-424b-bd4a-71a03f95d957\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb"
Sep 07 22:39:19.010740 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:19.010693    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht2kd\" (UniqueName: \"kubernetes.io/projected/98851745-710d-424b-bd4a-71a03f95d957-kube-api-access-ht2kd\") pod \"kuadrant-operator-controller-manager-6bd956ccf-cbqcb\" (UID: \"98851745-710d-424b-bd4a-71a03f95d957\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb"
Sep 07 22:39:19.011066 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:19.011044    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98851745-710d-424b-bd4a-71a03f95d957-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-cbqcb\" (UID: \"98851745-710d-424b-bd4a-71a03f95d957\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb"
Sep 07 22:39:19.025614 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:19.025582    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht2kd\" (UniqueName: \"kubernetes.io/projected/98851745-710d-424b-bd4a-71a03f95d957-kube-api-access-ht2kd\") pod \"kuadrant-operator-controller-manager-6bd956ccf-cbqcb\" (UID: \"98851745-710d-424b-bd4a-71a03f95d957\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb"
Sep 07 22:39:19.148087 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:19.148053    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb"
Sep 07 22:39:19.434378 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:19.434185    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb"]
Sep 07 22:39:19.438735 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:39:19.438689    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98851745_710d_424b_bd4a_71a03f95d957.slice/crio-d6309169240079d23bab5de4b655732523c70987ebf8d799ac37f4e459b3803b WatchSource:0}: Error finding container d6309169240079d23bab5de4b655732523c70987ebf8d799ac37f4e459b3803b: Status 404 returned error can't find the container with id d6309169240079d23bab5de4b655732523c70987ebf8d799ac37f4e459b3803b
Sep 07 22:39:19.566087 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:19.566047    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb" event={"ID":"98851745-710d-424b-bd4a-71a03f95d957","Type":"ContainerStarted","Data":"d6309169240079d23bab5de4b655732523c70987ebf8d799ac37f4e459b3803b"}
Sep 07 22:39:23.594889 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:23.594847    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb" event={"ID":"98851745-710d-424b-bd4a-71a03f95d957","Type":"ContainerStarted","Data":"e2b36ddd800f26be083d65a932dec4b0752325a1e69eca38be58f2dc7b28cac0"}
Sep 07 22:39:23.595363 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:23.594965    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb"
Sep 07 22:39:23.596812 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:23.596769    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-brb4b" podUID="8ae8a497-0cb2-4d45-8433-525235d960c4" containerName="manager" containerID="cri-o://bc98137324021e95f3ba6e63203d9ab2565d2d28a45ba9adb4997b0c7a348e1a" gracePeriod=2
Sep 07 22:39:23.615352 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:23.615281    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb" podStartSLOduration=1.663811959 podStartE2EDuration="5.615262256s" totalImagesPullingTime="3.951450297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:39:23.61352997 +0000 UTC m=+446.670362943" watchObservedRunningTime="2026-09-07 22:39:23.615262256 +0000 UTC m=+446.672095216"
Sep 07 22:39:23.840152 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:23.840122    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-brb4b"
Sep 07 22:39:23.842181 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:39:23.842146    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-brb4b\" is forbidden: User \"system:node:ip-10-0-138-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-99.ec2.internal' and this object" podUID="8ae8a497-0cb2-4d45-8433-525235d960c4" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-brb4b"
Sep 07 22:39:23.961433 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:23.961340    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmhdp\" (UniqueName: \"kubernetes.io/projected/8ae8a497-0cb2-4d45-8433-525235d960c4-kube-api-access-hmhdp\") pod \"8ae8a497-0cb2-4d45-8433-525235d960c4\" (UID: \"8ae8a497-0cb2-4d45-8433-525235d960c4\") "
Sep 07 22:39:23.961433 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:23.961394    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ae8a497-0cb2-4d45-8433-525235d960c4-extensions-socket-volume\") pod \"8ae8a497-0cb2-4d45-8433-525235d960c4\" (UID: \"8ae8a497-0cb2-4d45-8433-525235d960c4\") "
Sep 07 22:39:23.961750 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:23.961724    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae8a497-0cb2-4d45-8433-525235d960c4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8ae8a497-0cb2-4d45-8433-525235d960c4" (UID: "8ae8a497-0cb2-4d45-8433-525235d960c4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:39:23.963609 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:23.963578    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae8a497-0cb2-4d45-8433-525235d960c4-kube-api-access-hmhdp" (OuterVolumeSpecName: "kube-api-access-hmhdp") pod "8ae8a497-0cb2-4d45-8433-525235d960c4" (UID: "8ae8a497-0cb2-4d45-8433-525235d960c4"). InnerVolumeSpecName "kube-api-access-hmhdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:39:24.062104 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:24.062063    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmhdp\" (UniqueName: \"kubernetes.io/projected/8ae8a497-0cb2-4d45-8433-525235d960c4-kube-api-access-hmhdp\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:39:24.062104 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:24.062095    2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ae8a497-0cb2-4d45-8433-525235d960c4-extensions-socket-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:39:24.603160 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:24.603123    2567 generic.go:356] "Generic (PLEG): container finished" podID="8ae8a497-0cb2-4d45-8433-525235d960c4" containerID="bc98137324021e95f3ba6e63203d9ab2565d2d28a45ba9adb4997b0c7a348e1a" exitCode=2
Sep 07 22:39:24.603647 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:24.603166    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-brb4b"
Sep 07 22:39:24.603647 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:24.603226    2567 scope.go:117] "RemoveContainer" containerID="bc98137324021e95f3ba6e63203d9ab2565d2d28a45ba9adb4997b0c7a348e1a"
Sep 07 22:39:24.605370 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:39:24.605340    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-brb4b\" is forbidden: User \"system:node:ip-10-0-138-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-99.ec2.internal' and this object" podUID="8ae8a497-0cb2-4d45-8433-525235d960c4" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-brb4b"
Sep 07 22:39:24.612213 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:24.612194    2567 scope.go:117] "RemoveContainer" containerID="bc98137324021e95f3ba6e63203d9ab2565d2d28a45ba9adb4997b0c7a348e1a"
Sep 07 22:39:24.612522 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:39:24.612502    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc98137324021e95f3ba6e63203d9ab2565d2d28a45ba9adb4997b0c7a348e1a\": container with ID starting with bc98137324021e95f3ba6e63203d9ab2565d2d28a45ba9adb4997b0c7a348e1a not found: ID does not exist" containerID="bc98137324021e95f3ba6e63203d9ab2565d2d28a45ba9adb4997b0c7a348e1a"
Sep 07 22:39:24.612590 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:24.612539    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc98137324021e95f3ba6e63203d9ab2565d2d28a45ba9adb4997b0c7a348e1a"} err="failed to get container status \"bc98137324021e95f3ba6e63203d9ab2565d2d28a45ba9adb4997b0c7a348e1a\": rpc error: code = NotFound desc = could not find container \"bc98137324021e95f3ba6e63203d9ab2565d2d28a45ba9adb4997b0c7a348e1a\": container with ID starting with bc98137324021e95f3ba6e63203d9ab2565d2d28a45ba9adb4997b0c7a348e1a not found: ID does not exist"
Sep 07 22:39:24.613677 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:39:24.613654    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-brb4b\" is forbidden: User \"system:node:ip-10-0-138-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-99.ec2.internal' and this object" podUID="8ae8a497-0cb2-4d45-8433-525235d960c4" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-brb4b"
Sep 07 22:39:25.465139 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:25.465102    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ae8a497-0cb2-4d45-8433-525235d960c4" path="/var/lib/kubelet/pods/8ae8a497-0cb2-4d45-8433-525235d960c4/volumes"
Sep 07 22:39:34.606407 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:34.606372    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb"
Sep 07 22:39:51.011487 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:51.011452    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb"]
Sep 07 22:39:51.011982 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:51.011696    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb" podUID="98851745-710d-424b-bd4a-71a03f95d957" containerName="manager" containerID="cri-o://e2b36ddd800f26be083d65a932dec4b0752325a1e69eca38be58f2dc7b28cac0" gracePeriod=10
Sep 07 22:39:51.265131 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:51.265064    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb"
Sep 07 22:39:51.268064 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:51.268032    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98851745-710d-424b-bd4a-71a03f95d957-extensions-socket-volume\") pod \"98851745-710d-424b-bd4a-71a03f95d957\" (UID: \"98851745-710d-424b-bd4a-71a03f95d957\") "
Sep 07 22:39:51.268196 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:51.268071    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ht2kd\" (UniqueName: \"kubernetes.io/projected/98851745-710d-424b-bd4a-71a03f95d957-kube-api-access-ht2kd\") pod \"98851745-710d-424b-bd4a-71a03f95d957\" (UID: \"98851745-710d-424b-bd4a-71a03f95d957\") "
Sep 07 22:39:51.268522 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:51.268497    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98851745-710d-424b-bd4a-71a03f95d957-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "98851745-710d-424b-bd4a-71a03f95d957" (UID: "98851745-710d-424b-bd4a-71a03f95d957"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:39:51.270186 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:51.270159    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98851745-710d-424b-bd4a-71a03f95d957-kube-api-access-ht2kd" (OuterVolumeSpecName: "kube-api-access-ht2kd") pod "98851745-710d-424b-bd4a-71a03f95d957" (UID: "98851745-710d-424b-bd4a-71a03f95d957"). InnerVolumeSpecName "kube-api-access-ht2kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:39:51.368897 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:51.368861    2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98851745-710d-424b-bd4a-71a03f95d957-extensions-socket-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:39:51.368897 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:51.368892    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht2kd\" (UniqueName: \"kubernetes.io/projected/98851745-710d-424b-bd4a-71a03f95d957-kube-api-access-ht2kd\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:39:51.731088 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:51.731049    2567 generic.go:356] "Generic (PLEG): container finished" podID="98851745-710d-424b-bd4a-71a03f95d957" containerID="e2b36ddd800f26be083d65a932dec4b0752325a1e69eca38be58f2dc7b28cac0" exitCode=0
Sep 07 22:39:51.731285 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:51.731097    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb"
Sep 07 22:39:51.731285 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:51.731135    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb" event={"ID":"98851745-710d-424b-bd4a-71a03f95d957","Type":"ContainerDied","Data":"e2b36ddd800f26be083d65a932dec4b0752325a1e69eca38be58f2dc7b28cac0"}
Sep 07 22:39:51.731285 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:51.731193    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb" event={"ID":"98851745-710d-424b-bd4a-71a03f95d957","Type":"ContainerDied","Data":"d6309169240079d23bab5de4b655732523c70987ebf8d799ac37f4e459b3803b"}
Sep 07 22:39:51.731285 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:51.731214    2567 scope.go:117] "RemoveContainer" containerID="e2b36ddd800f26be083d65a932dec4b0752325a1e69eca38be58f2dc7b28cac0"
Sep 07 22:39:51.740386 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:51.740362    2567 scope.go:117] "RemoveContainer" containerID="e2b36ddd800f26be083d65a932dec4b0752325a1e69eca38be58f2dc7b28cac0"
Sep 07 22:39:51.740753 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:39:51.740729    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b36ddd800f26be083d65a932dec4b0752325a1e69eca38be58f2dc7b28cac0\": container with ID starting with e2b36ddd800f26be083d65a932dec4b0752325a1e69eca38be58f2dc7b28cac0 not found: ID does not exist" containerID="e2b36ddd800f26be083d65a932dec4b0752325a1e69eca38be58f2dc7b28cac0"
Sep 07 22:39:51.740821 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:51.740761    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b36ddd800f26be083d65a932dec4b0752325a1e69eca38be58f2dc7b28cac0"} err="failed to get container status \"e2b36ddd800f26be083d65a932dec4b0752325a1e69eca38be58f2dc7b28cac0\": rpc error: code = NotFound desc = could not find container \"e2b36ddd800f26be083d65a932dec4b0752325a1e69eca38be58f2dc7b28cac0\": container with ID starting with e2b36ddd800f26be083d65a932dec4b0752325a1e69eca38be58f2dc7b28cac0 not found: ID does not exist"
Sep 07 22:39:51.746830 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:51.746797    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb"]
Sep 07 22:39:51.757688 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:51.757656    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cbqcb"]
Sep 07 22:39:53.464955 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:39:53.464910    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98851745-710d-424b-bd4a-71a03f95d957" path="/var/lib/kubelet/pods/98851745-710d-424b-bd4a-71a03f95d957/volumes"
Sep 07 22:40:07.660922 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.658441    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-dt8m4"]
Sep 07 22:40:07.660922 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.659154    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ae8a497-0cb2-4d45-8433-525235d960c4" containerName="manager"
Sep 07 22:40:07.660922 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.659176    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ae8a497-0cb2-4d45-8433-525235d960c4" containerName="manager"
Sep 07 22:40:07.660922 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.659199    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98851745-710d-424b-bd4a-71a03f95d957" containerName="manager"
Sep 07 22:40:07.660922 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.659208    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="98851745-710d-424b-bd4a-71a03f95d957" containerName="manager"
Sep 07 22:40:07.660922 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.659409    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ae8a497-0cb2-4d45-8433-525235d960c4" containerName="manager"
Sep 07 22:40:07.660922 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.659429    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="98851745-710d-424b-bd4a-71a03f95d957" containerName="manager"
Sep 07 22:40:07.664688 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.664661    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-dt8m4"
Sep 07 22:40:07.666870 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.666843    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 07 22:40:07.667275 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.667233    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 07 22:40:07.667448 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.667419    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 07 22:40:07.667699 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.667676    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nwtjv\""
Sep 07 22:40:07.667934 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.667911    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-dt8m4"]
Sep 07 22:40:07.703736 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.703698    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh5vl\" (UniqueName: \"kubernetes.io/projected/82623fc3-a763-484b-89cb-4c860331aa10-kube-api-access-vh5vl\") pod \"limitador-limitador-859c4d96d6-dt8m4\" (UID: \"82623fc3-a763-484b-89cb-4c860331aa10\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-dt8m4"
Sep 07 22:40:07.703911 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.703749    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/82623fc3-a763-484b-89cb-4c860331aa10-config-file\") pod \"limitador-limitador-859c4d96d6-dt8m4\" (UID: \"82623fc3-a763-484b-89cb-4c860331aa10\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-dt8m4"
Sep 07 22:40:07.755074 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.755039    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-dt8m4"]
Sep 07 22:40:07.804964 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.804926    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/82623fc3-a763-484b-89cb-4c860331aa10-config-file\") pod \"limitador-limitador-859c4d96d6-dt8m4\" (UID: \"82623fc3-a763-484b-89cb-4c860331aa10\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-dt8m4"
Sep 07 22:40:07.805133 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.805021    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh5vl\" (UniqueName: \"kubernetes.io/projected/82623fc3-a763-484b-89cb-4c860331aa10-kube-api-access-vh5vl\") pod \"limitador-limitador-859c4d96d6-dt8m4\" (UID: \"82623fc3-a763-484b-89cb-4c860331aa10\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-dt8m4"
Sep 07 22:40:07.805609 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.805589    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/82623fc3-a763-484b-89cb-4c860331aa10-config-file\") pod \"limitador-limitador-859c4d96d6-dt8m4\" (UID: \"82623fc3-a763-484b-89cb-4c860331aa10\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-dt8m4"
Sep 07 22:40:07.812357 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.812329    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh5vl\" (UniqueName: \"kubernetes.io/projected/82623fc3-a763-484b-89cb-4c860331aa10-kube-api-access-vh5vl\") pod \"limitador-limitador-859c4d96d6-dt8m4\" (UID: \"82623fc3-a763-484b-89cb-4c860331aa10\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-dt8m4"
Sep 07 22:40:07.976225 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:07.976124    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-dt8m4"
Sep 07 22:40:08.246294 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:08.246193    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-dt8m4"]
Sep 07 22:40:08.248993 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:40:08.248962    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82623fc3_a763_484b_89cb_4c860331aa10.slice/crio-73345064247e187422246ac617b7710e05d9bfee4ac7ae76c91853359af07d9f WatchSource:0}: Error finding container 73345064247e187422246ac617b7710e05d9bfee4ac7ae76c91853359af07d9f: Status 404 returned error can't find the container with id 73345064247e187422246ac617b7710e05d9bfee4ac7ae76c91853359af07d9f
Sep 07 22:40:08.432911 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:08.432877    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-dnb5z"]
Sep 07 22:40:08.437340 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:08.437322    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-dnb5z"
Sep 07 22:40:08.439678 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:08.439653    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sdmsr\""
Sep 07 22:40:08.443661 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:08.443633    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-dnb5z"]
Sep 07 22:40:08.511778 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:08.511675    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whsgw\" (UniqueName: \"kubernetes.io/projected/2a276f33-2306-4a78-9dc4-3ad05fc18116-kube-api-access-whsgw\") pod \"authorino-b54df94b4-dnb5z\" (UID: \"2a276f33-2306-4a78-9dc4-3ad05fc18116\") " pod="kuadrant-system/authorino-b54df94b4-dnb5z"
Sep 07 22:40:08.612841 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:08.612801    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whsgw\" (UniqueName: \"kubernetes.io/projected/2a276f33-2306-4a78-9dc4-3ad05fc18116-kube-api-access-whsgw\") pod \"authorino-b54df94b4-dnb5z\" (UID: \"2a276f33-2306-4a78-9dc4-3ad05fc18116\") " pod="kuadrant-system/authorino-b54df94b4-dnb5z"
Sep 07 22:40:08.620301 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:08.620277    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whsgw\" (UniqueName: \"kubernetes.io/projected/2a276f33-2306-4a78-9dc4-3ad05fc18116-kube-api-access-whsgw\") pod \"authorino-b54df94b4-dnb5z\" (UID: \"2a276f33-2306-4a78-9dc4-3ad05fc18116\") " pod="kuadrant-system/authorino-b54df94b4-dnb5z"
Sep 07 22:40:08.747808 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:08.747766    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-dnb5z"
Sep 07 22:40:08.809859 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:08.809824    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-dt8m4" event={"ID":"82623fc3-a763-484b-89cb-4c860331aa10","Type":"ContainerStarted","Data":"73345064247e187422246ac617b7710e05d9bfee4ac7ae76c91853359af07d9f"}
Sep 07 22:40:09.015675 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:09.015640    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-dnb5z"]
Sep 07 22:40:09.020895 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:40:09.020853    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a276f33_2306_4a78_9dc4_3ad05fc18116.slice/crio-20409e579db9acde7fbd235ccb8ed299abfcd8d69712d415047bf6b40f36b56a WatchSource:0}: Error finding container 20409e579db9acde7fbd235ccb8ed299abfcd8d69712d415047bf6b40f36b56a: Status 404 returned error can't find the container with id 20409e579db9acde7fbd235ccb8ed299abfcd8d69712d415047bf6b40f36b56a
Sep 07 22:40:09.816255 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:09.816203    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-dnb5z" event={"ID":"2a276f33-2306-4a78-9dc4-3ad05fc18116","Type":"ContainerStarted","Data":"20409e579db9acde7fbd235ccb8ed299abfcd8d69712d415047bf6b40f36b56a"}
Sep 07 22:40:11.828826 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:11.828777    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-dnb5z" event={"ID":"2a276f33-2306-4a78-9dc4-3ad05fc18116","Type":"ContainerStarted","Data":"ba2047ca5e752603831db13df8c5861a6cbdd20cc6ed84fbcce02bf98bff84e5"}
Sep 07 22:40:11.842789 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:11.842731    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-dnb5z" podStartSLOduration=1.531983657 podStartE2EDuration="3.842713943s" totalImagesPullingTime="2.310730286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:40:11.841552145 +0000 UTC m=+494.898385105" watchObservedRunningTime="2026-09-07 22:40:11.842713943 +0000 UTC m=+494.899546904"
Sep 07 22:40:12.153719 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:12.153632    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-dnb5z"]
Sep 07 22:40:13.837338 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:13.837294    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-dnb5z" podUID="2a276f33-2306-4a78-9dc4-3ad05fc18116" containerName="authorino" containerID="cri-o://ba2047ca5e752603831db13df8c5861a6cbdd20cc6ed84fbcce02bf98bff84e5" gracePeriod=30
Sep 07 22:40:14.085123 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:14.085096    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-dnb5z"
Sep 07 22:40:14.159810 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:14.159710    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whsgw\" (UniqueName: \"kubernetes.io/projected/2a276f33-2306-4a78-9dc4-3ad05fc18116-kube-api-access-whsgw\") pod \"2a276f33-2306-4a78-9dc4-3ad05fc18116\" (UID: \"2a276f33-2306-4a78-9dc4-3ad05fc18116\") "
Sep 07 22:40:14.161811 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:14.161785    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a276f33-2306-4a78-9dc4-3ad05fc18116-kube-api-access-whsgw" (OuterVolumeSpecName: "kube-api-access-whsgw") pod "2a276f33-2306-4a78-9dc4-3ad05fc18116" (UID: "2a276f33-2306-4a78-9dc4-3ad05fc18116"). InnerVolumeSpecName "kube-api-access-whsgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:40:14.260822 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:14.260779    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whsgw\" (UniqueName: \"kubernetes.io/projected/2a276f33-2306-4a78-9dc4-3ad05fc18116-kube-api-access-whsgw\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:40:14.843946 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:14.843909    2567 generic.go:356] "Generic (PLEG): container finished" podID="2a276f33-2306-4a78-9dc4-3ad05fc18116" containerID="ba2047ca5e752603831db13df8c5861a6cbdd20cc6ed84fbcce02bf98bff84e5" exitCode=0
Sep 07 22:40:14.844375 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:14.843954    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-dnb5z"
Sep 07 22:40:14.844375 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:14.843995    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-dnb5z" event={"ID":"2a276f33-2306-4a78-9dc4-3ad05fc18116","Type":"ContainerDied","Data":"ba2047ca5e752603831db13df8c5861a6cbdd20cc6ed84fbcce02bf98bff84e5"}
Sep 07 22:40:14.844375 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:14.844043    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-dnb5z" event={"ID":"2a276f33-2306-4a78-9dc4-3ad05fc18116","Type":"ContainerDied","Data":"20409e579db9acde7fbd235ccb8ed299abfcd8d69712d415047bf6b40f36b56a"}
Sep 07 22:40:14.844375 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:14.844065    2567 scope.go:117] "RemoveContainer" containerID="ba2047ca5e752603831db13df8c5861a6cbdd20cc6ed84fbcce02bf98bff84e5"
Sep 07 22:40:14.852731 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:14.852712    2567 scope.go:117] "RemoveContainer" containerID="ba2047ca5e752603831db13df8c5861a6cbdd20cc6ed84fbcce02bf98bff84e5"
Sep 07 22:40:14.853074 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:40:14.853052    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2047ca5e752603831db13df8c5861a6cbdd20cc6ed84fbcce02bf98bff84e5\": container with ID starting with ba2047ca5e752603831db13df8c5861a6cbdd20cc6ed84fbcce02bf98bff84e5 not found: ID does not exist" containerID="ba2047ca5e752603831db13df8c5861a6cbdd20cc6ed84fbcce02bf98bff84e5"
Sep 07 22:40:14.853123 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:14.853084    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2047ca5e752603831db13df8c5861a6cbdd20cc6ed84fbcce02bf98bff84e5"} err="failed to get container status \"ba2047ca5e752603831db13df8c5861a6cbdd20cc6ed84fbcce02bf98bff84e5\": rpc error: code = NotFound desc = could not find container \"ba2047ca5e752603831db13df8c5861a6cbdd20cc6ed84fbcce02bf98bff84e5\": container with ID starting with ba2047ca5e752603831db13df8c5861a6cbdd20cc6ed84fbcce02bf98bff84e5 not found: ID does not exist"
Sep 07 22:40:14.864038 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:14.864007    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-dnb5z"]
Sep 07 22:40:14.867802 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:14.867775    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-dnb5z"]
Sep 07 22:40:15.465326 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:15.465294    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a276f33-2306-4a78-9dc4-3ad05fc18116" path="/var/lib/kubelet/pods/2a276f33-2306-4a78-9dc4-3ad05fc18116/volumes"
Sep 07 22:40:17.867129 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:17.867089    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-dt8m4" event={"ID":"82623fc3-a763-484b-89cb-4c860331aa10","Type":"ContainerStarted","Data":"5189ef2da560070cdd22342262d6d8b6c5bdeb389d1e7af3fa85b895256f97c3"}
Sep 07 22:40:17.867597 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:17.867215    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-dt8m4"
Sep 07 22:40:17.882659 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:17.882605    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-dt8m4" podStartSLOduration=1.675718076 podStartE2EDuration="10.882587979s" totalImagesPullingTime="9.206869903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:40:17.881061142 +0000 UTC m=+500.937894105" watchObservedRunningTime="2026-09-07 22:40:17.882587979 +0000 UTC m=+500.939420940"
Sep 07 22:40:21.125796 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:21.125761    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-dt8m4"]
Sep 07 22:40:21.126281 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:21.125939    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-dt8m4" podUID="82623fc3-a763-484b-89cb-4c860331aa10" containerName="limitador" containerID="cri-o://5189ef2da560070cdd22342262d6d8b6c5bdeb389d1e7af3fa85b895256f97c3" gracePeriod=30
Sep 07 22:40:21.888318 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:21.888281    2567 generic.go:356] "Generic (PLEG): container finished" podID="82623fc3-a763-484b-89cb-4c860331aa10" containerID="5189ef2da560070cdd22342262d6d8b6c5bdeb389d1e7af3fa85b895256f97c3" exitCode=0
Sep 07 22:40:21.888567 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:21.888355    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-dt8m4" event={"ID":"82623fc3-a763-484b-89cb-4c860331aa10","Type":"ContainerDied","Data":"5189ef2da560070cdd22342262d6d8b6c5bdeb389d1e7af3fa85b895256f97c3"}
Sep 07 22:40:22.078612 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:22.078587    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-dt8m4"
Sep 07 22:40:22.231879 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:22.231786    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vh5vl\" (UniqueName: \"kubernetes.io/projected/82623fc3-a763-484b-89cb-4c860331aa10-kube-api-access-vh5vl\") pod \"82623fc3-a763-484b-89cb-4c860331aa10\" (UID: \"82623fc3-a763-484b-89cb-4c860331aa10\") "
Sep 07 22:40:22.231879 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:22.231829    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/82623fc3-a763-484b-89cb-4c860331aa10-config-file\") pod \"82623fc3-a763-484b-89cb-4c860331aa10\" (UID: \"82623fc3-a763-484b-89cb-4c860331aa10\") "
Sep 07 22:40:22.232411 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:22.232272    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82623fc3-a763-484b-89cb-4c860331aa10-config-file" (OuterVolumeSpecName: "config-file") pod "82623fc3-a763-484b-89cb-4c860331aa10" (UID: "82623fc3-a763-484b-89cb-4c860331aa10"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:40:22.234073 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:22.234032    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82623fc3-a763-484b-89cb-4c860331aa10-kube-api-access-vh5vl" (OuterVolumeSpecName: "kube-api-access-vh5vl") pod "82623fc3-a763-484b-89cb-4c860331aa10" (UID: "82623fc3-a763-484b-89cb-4c860331aa10"). InnerVolumeSpecName "kube-api-access-vh5vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:40:22.332632 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:22.332594    2567 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/82623fc3-a763-484b-89cb-4c860331aa10-config-file\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:40:22.332632 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:22.332626    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh5vl\" (UniqueName: \"kubernetes.io/projected/82623fc3-a763-484b-89cb-4c860331aa10-kube-api-access-vh5vl\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:40:22.894041 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:22.894005    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-dt8m4"
Sep 07 22:40:22.894212 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:22.894014    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-dt8m4" event={"ID":"82623fc3-a763-484b-89cb-4c860331aa10","Type":"ContainerDied","Data":"73345064247e187422246ac617b7710e05d9bfee4ac7ae76c91853359af07d9f"}
Sep 07 22:40:22.894212 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:22.894179    2567 scope.go:117] "RemoveContainer" containerID="5189ef2da560070cdd22342262d6d8b6c5bdeb389d1e7af3fa85b895256f97c3"
Sep 07 22:40:22.914691 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:22.914658    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-dt8m4"]
Sep 07 22:40:22.916444 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:22.916420    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-dt8m4"]
Sep 07 22:40:23.465177 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:23.465144    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82623fc3-a763-484b-89cb-4c860331aa10" path="/var/lib/kubelet/pods/82623fc3-a763-484b-89cb-4c860331aa10/volumes"
Sep 07 22:40:24.835269 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:24.835220    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-tz2vq"]
Sep 07 22:40:24.835735 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:24.835718    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82623fc3-a763-484b-89cb-4c860331aa10" containerName="limitador"
Sep 07 22:40:24.835782 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:24.835738    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="82623fc3-a763-484b-89cb-4c860331aa10" containerName="limitador"
Sep 07 22:40:24.835782 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:24.835764    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a276f33-2306-4a78-9dc4-3ad05fc18116" containerName="authorino"
Sep 07 22:40:24.835782 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:24.835774    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a276f33-2306-4a78-9dc4-3ad05fc18116" containerName="authorino"
Sep 07 22:40:24.835881 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:24.835867    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a276f33-2306-4a78-9dc4-3ad05fc18116" containerName="authorino"
Sep 07 22:40:24.835919 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:24.835879    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="82623fc3-a763-484b-89cb-4c860331aa10" containerName="limitador"
Sep 07 22:40:24.840618 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:24.840595    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tz2vq"
Sep 07 22:40:24.842788 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:24.842754    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-dbcg5\""
Sep 07 22:40:24.842788 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:24.842781    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 07 22:40:24.842961 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:24.842853    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 07 22:40:24.843352 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:24.843334    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:40:24.846390 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:24.846366    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-tz2vq"]
Sep 07 22:40:24.953434 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:24.953391    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwswp\" (UniqueName: \"kubernetes.io/projected/341581c6-0c54-443b-a949-78b942f80c5b-kube-api-access-wwswp\") pod \"postgres-69fb4594cf-tz2vq\" (UID: \"341581c6-0c54-443b-a949-78b942f80c5b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tz2vq"
Sep 07 22:40:24.953601 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:24.953451    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/341581c6-0c54-443b-a949-78b942f80c5b-data\") pod \"postgres-69fb4594cf-tz2vq\" (UID: \"341581c6-0c54-443b-a949-78b942f80c5b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tz2vq"
Sep 07 22:40:25.054414 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:25.054371    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwswp\" (UniqueName: \"kubernetes.io/projected/341581c6-0c54-443b-a949-78b942f80c5b-kube-api-access-wwswp\") pod \"postgres-69fb4594cf-tz2vq\" (UID: \"341581c6-0c54-443b-a949-78b942f80c5b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tz2vq"
Sep 07 22:40:25.054588 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:25.054426    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/341581c6-0c54-443b-a949-78b942f80c5b-data\") pod \"postgres-69fb4594cf-tz2vq\" (UID: \"341581c6-0c54-443b-a949-78b942f80c5b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tz2vq"
Sep 07 22:40:25.054756 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:25.054741    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/341581c6-0c54-443b-a949-78b942f80c5b-data\") pod \"postgres-69fb4594cf-tz2vq\" (UID: \"341581c6-0c54-443b-a949-78b942f80c5b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tz2vq"
Sep 07 22:40:25.061609 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:25.061581    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwswp\" (UniqueName: \"kubernetes.io/projected/341581c6-0c54-443b-a949-78b942f80c5b-kube-api-access-wwswp\") pod \"postgres-69fb4594cf-tz2vq\" (UID: \"341581c6-0c54-443b-a949-78b942f80c5b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tz2vq"
Sep 07 22:40:25.152220 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:25.152138    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tz2vq"
Sep 07 22:40:25.424958 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:25.424930    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-tz2vq"]
Sep 07 22:40:25.427300 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:40:25.427273    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341581c6_0c54_443b_a949_78b942f80c5b.slice/crio-ab6067694de2116c86390114a8d07d282bd9f8e28b519c95fc1c8b05bc633afc WatchSource:0}: Error finding container ab6067694de2116c86390114a8d07d282bd9f8e28b519c95fc1c8b05bc633afc: Status 404 returned error can't find the container with id ab6067694de2116c86390114a8d07d282bd9f8e28b519c95fc1c8b05bc633afc
Sep 07 22:40:25.909835 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:25.909804    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tz2vq" event={"ID":"341581c6-0c54-443b-a949-78b942f80c5b","Type":"ContainerStarted","Data":"ab6067694de2116c86390114a8d07d282bd9f8e28b519c95fc1c8b05bc633afc"}
Sep 07 22:40:30.937190 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:30.937150    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tz2vq" event={"ID":"341581c6-0c54-443b-a949-78b942f80c5b","Type":"ContainerStarted","Data":"0a2813e78fa572df3703160bcc930f77a17a9bb3566282239008b0fe1df64e4d"}
Sep 07 22:40:30.937581 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:30.937275    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tz2vq"
Sep 07 22:40:30.950332 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:30.950216    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tz2vq" podStartSLOduration=2.089908538 podStartE2EDuration="6.950189423s" totalImagesPullingTime="4.860280885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:40:30.950109898 +0000 UTC m=+514.006942859" watchObservedRunningTime="2026-09-07 22:40:30.950189423 +0000 UTC m=+514.007022417"
Sep 07 22:40:36.970722 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:36.970688    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tz2vq"
Sep 07 22:40:37.498503 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:37.498473    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-qg26t"]
Sep 07 22:40:37.504147 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:37.504123    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-qg26t"
Sep 07 22:40:37.507435 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:37.507412    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sdmsr\""
Sep 07 22:40:37.507555 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:37.507469    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 07 22:40:37.507779 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:37.507763    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 07 22:40:37.522058 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:37.522028    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-qg26t"]
Sep 07 22:40:37.669669 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:37.669632    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n7r7\" (UniqueName: \"kubernetes.io/projected/f0051b33-86ef-4a21-92cf-3b067088e83c-kube-api-access-2n7r7\") pod \"authorino-5dc7ccb5f-qg26t\" (UID: \"f0051b33-86ef-4a21-92cf-3b067088e83c\") " pod="kuadrant-system/authorino-5dc7ccb5f-qg26t"
Sep 07 22:40:37.697572 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:37.697535    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-qg26t"]
Sep 07 22:40:37.697823 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:40:37.697803    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2n7r7], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-qg26t" podUID="f0051b33-86ef-4a21-92cf-3b067088e83c"
Sep 07 22:40:37.723674 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:37.723641    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74b854cf7b-bmgdk"]
Sep 07 22:40:37.726647 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:37.726629    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74b854cf7b-bmgdk"
Sep 07 22:40:37.731085 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:37.731061    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74b854cf7b-bmgdk"]
Sep 07 22:40:37.770868 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:37.770774    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n7r7\" (UniqueName: \"kubernetes.io/projected/f0051b33-86ef-4a21-92cf-3b067088e83c-kube-api-access-2n7r7\") pod \"authorino-5dc7ccb5f-qg26t\" (UID: \"f0051b33-86ef-4a21-92cf-3b067088e83c\") " pod="kuadrant-system/authorino-5dc7ccb5f-qg26t"
Sep 07 22:40:37.778300 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:37.778268    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n7r7\" (UniqueName: \"kubernetes.io/projected/f0051b33-86ef-4a21-92cf-3b067088e83c-kube-api-access-2n7r7\") pod \"authorino-5dc7ccb5f-qg26t\" (UID: \"f0051b33-86ef-4a21-92cf-3b067088e83c\") " pod="kuadrant-system/authorino-5dc7ccb5f-qg26t"
Sep 07 22:40:37.872115 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:37.872070    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq7qp\" (UniqueName: \"kubernetes.io/projected/67f39360-19f3-4f04-90fc-dc23223b65ea-kube-api-access-bq7qp\") pod \"authorino-74b854cf7b-bmgdk\" (UID: \"67f39360-19f3-4f04-90fc-dc23223b65ea\") " pod="kuadrant-system/authorino-74b854cf7b-bmgdk"
Sep 07 22:40:37.968978 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:37.968946    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-qg26t"
Sep 07 22:40:37.973567 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:37.973542    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq7qp\" (UniqueName: \"kubernetes.io/projected/67f39360-19f3-4f04-90fc-dc23223b65ea-kube-api-access-bq7qp\") pod \"authorino-74b854cf7b-bmgdk\" (UID: \"67f39360-19f3-4f04-90fc-dc23223b65ea\") " pod="kuadrant-system/authorino-74b854cf7b-bmgdk"
Sep 07 22:40:37.976320 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:37.976300    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-qg26t"
Sep 07 22:40:37.980470 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:37.980446    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq7qp\" (UniqueName: \"kubernetes.io/projected/67f39360-19f3-4f04-90fc-dc23223b65ea-kube-api-access-bq7qp\") pod \"authorino-74b854cf7b-bmgdk\" (UID: \"67f39360-19f3-4f04-90fc-dc23223b65ea\") " pod="kuadrant-system/authorino-74b854cf7b-bmgdk"
Sep 07 22:40:38.036407 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:38.036321    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74b854cf7b-bmgdk"
Sep 07 22:40:38.074131 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:38.074088    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n7r7\" (UniqueName: \"kubernetes.io/projected/f0051b33-86ef-4a21-92cf-3b067088e83c-kube-api-access-2n7r7\") pod \"f0051b33-86ef-4a21-92cf-3b067088e83c\" (UID: \"f0051b33-86ef-4a21-92cf-3b067088e83c\") "
Sep 07 22:40:38.076206 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:38.076168    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0051b33-86ef-4a21-92cf-3b067088e83c-kube-api-access-2n7r7" (OuterVolumeSpecName: "kube-api-access-2n7r7") pod "f0051b33-86ef-4a21-92cf-3b067088e83c" (UID: "f0051b33-86ef-4a21-92cf-3b067088e83c"). InnerVolumeSpecName "kube-api-access-2n7r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:40:38.174804 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:38.174759    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n7r7\" (UniqueName: \"kubernetes.io/projected/f0051b33-86ef-4a21-92cf-3b067088e83c-kube-api-access-2n7r7\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:40:38.306617 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:38.306579    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74b854cf7b-bmgdk"]
Sep 07 22:40:38.309255 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:40:38.309211    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67f39360_19f3_4f04_90fc_dc23223b65ea.slice/crio-11b242134735129a6372ac037a1bdeb496c6b02e75d57113f24670f742ee469c WatchSource:0}: Error finding container 11b242134735129a6372ac037a1bdeb496c6b02e75d57113f24670f742ee469c: Status 404 returned error can't find the container with id 11b242134735129a6372ac037a1bdeb496c6b02e75d57113f24670f742ee469c
Sep 07 22:40:38.975381 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:38.975286    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74b854cf7b-bmgdk" event={"ID":"67f39360-19f3-4f04-90fc-dc23223b65ea","Type":"ContainerStarted","Data":"05e50d4a656c9d4619f776ab22b4654ed0a878c188982eebc523d521f57dfed4"}
Sep 07 22:40:38.975381 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:38.975332    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-qg26t"
Sep 07 22:40:38.975850 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:38.975338    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74b854cf7b-bmgdk" event={"ID":"67f39360-19f3-4f04-90fc-dc23223b65ea","Type":"ContainerStarted","Data":"11b242134735129a6372ac037a1bdeb496c6b02e75d57113f24670f742ee469c"}
Sep 07 22:40:38.988079 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:38.988032    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-74b854cf7b-bmgdk" podStartSLOduration=1.713787427 podStartE2EDuration="1.988013848s" totalImagesPullingTime="274.226421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:40:38.986709904 +0000 UTC m=+522.043542866" watchObservedRunningTime="2026-09-07 22:40:38.988013848 +0000 UTC m=+522.044846810"
Sep 07 22:40:39.007498 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:39.007462    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-qg26t"]
Sep 07 22:40:39.016517 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:39.016476    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-qg26t"]
Sep 07 22:40:39.081107 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:39.081067    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f846f8c86-jc7fb"]
Sep 07 22:40:39.086905 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:39.086872    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f846f8c86-jc7fb"
Sep 07 22:40:39.089507 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:39.089459    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 07 22:40:39.090678 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:39.090650    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f846f8c86-jc7fb"]
Sep 07 22:40:39.285936 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:39.285839    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/82fb8c01-058c-4b5e-b3c1-baa2002242b2-tls-cert\") pod \"authorino-6f846f8c86-jc7fb\" (UID: \"82fb8c01-058c-4b5e-b3c1-baa2002242b2\") " pod="kuadrant-system/authorino-6f846f8c86-jc7fb"
Sep 07 22:40:39.286081 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:39.286053    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtnfj\" (UniqueName: \"kubernetes.io/projected/82fb8c01-058c-4b5e-b3c1-baa2002242b2-kube-api-access-jtnfj\") pod \"authorino-6f846f8c86-jc7fb\" (UID: \"82fb8c01-058c-4b5e-b3c1-baa2002242b2\") " pod="kuadrant-system/authorino-6f846f8c86-jc7fb"
Sep 07 22:40:39.387150 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:39.387112    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnfj\" (UniqueName: \"kubernetes.io/projected/82fb8c01-058c-4b5e-b3c1-baa2002242b2-kube-api-access-jtnfj\") pod \"authorino-6f846f8c86-jc7fb\" (UID: \"82fb8c01-058c-4b5e-b3c1-baa2002242b2\") " pod="kuadrant-system/authorino-6f846f8c86-jc7fb"
Sep 07 22:40:39.387150 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:39.387158    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/82fb8c01-058c-4b5e-b3c1-baa2002242b2-tls-cert\") pod \"authorino-6f846f8c86-jc7fb\" (UID: \"82fb8c01-058c-4b5e-b3c1-baa2002242b2\") " pod="kuadrant-system/authorino-6f846f8c86-jc7fb"
Sep 07 22:40:39.389616 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:39.389592    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/82fb8c01-058c-4b5e-b3c1-baa2002242b2-tls-cert\") pod \"authorino-6f846f8c86-jc7fb\" (UID: \"82fb8c01-058c-4b5e-b3c1-baa2002242b2\") " pod="kuadrant-system/authorino-6f846f8c86-jc7fb"
Sep 07 22:40:39.394057 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:39.394034    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtnfj\" (UniqueName: \"kubernetes.io/projected/82fb8c01-058c-4b5e-b3c1-baa2002242b2-kube-api-access-jtnfj\") pod \"authorino-6f846f8c86-jc7fb\" (UID: \"82fb8c01-058c-4b5e-b3c1-baa2002242b2\") " pod="kuadrant-system/authorino-6f846f8c86-jc7fb"
Sep 07 22:40:39.398888 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:39.398855    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f846f8c86-jc7fb"
Sep 07 22:40:39.466062 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:39.466029    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0051b33-86ef-4a21-92cf-3b067088e83c" path="/var/lib/kubelet/pods/f0051b33-86ef-4a21-92cf-3b067088e83c/volumes"
Sep 07 22:40:39.665075 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:39.664975    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f846f8c86-jc7fb"]
Sep 07 22:40:39.667699 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:40:39.667672    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82fb8c01_058c_4b5e_b3c1_baa2002242b2.slice/crio-f53735bd028477f84b0b0152f39702a57a059870024f18d18d9deb49a826a578 WatchSource:0}: Error finding container f53735bd028477f84b0b0152f39702a57a059870024f18d18d9deb49a826a578: Status 404 returned error can't find the container with id f53735bd028477f84b0b0152f39702a57a059870024f18d18d9deb49a826a578
Sep 07 22:40:39.981371 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:39.981269    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f846f8c86-jc7fb" event={"ID":"82fb8c01-058c-4b5e-b3c1-baa2002242b2","Type":"ContainerStarted","Data":"f53735bd028477f84b0b0152f39702a57a059870024f18d18d9deb49a826a578"}
Sep 07 22:40:40.988173 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:40.988131    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f846f8c86-jc7fb" event={"ID":"82fb8c01-058c-4b5e-b3c1-baa2002242b2","Type":"ContainerStarted","Data":"856c789add86ceaf66819c6955eab40813352cc07fe7502c1079e096f713e0a4"}
Sep 07 22:40:41.001639 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:41.001585    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6f846f8c86-jc7fb" podStartSLOduration=1.5975122339999999 podStartE2EDuration="2.001567606s" totalImagesPullingTime="404.055372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:40:41.00078119 +0000 UTC m=+524.057614176" watchObservedRunningTime="2026-09-07 22:40:41.001567606 +0000 UTC m=+524.058400567"
Sep 07 22:40:41.025239 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:41.025193    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74b854cf7b-bmgdk"]
Sep 07 22:40:41.025431 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:41.025402    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-74b854cf7b-bmgdk" podUID="67f39360-19f3-4f04-90fc-dc23223b65ea" containerName="authorino" containerID="cri-o://05e50d4a656c9d4619f776ab22b4654ed0a878c188982eebc523d521f57dfed4" gracePeriod=30
Sep 07 22:40:41.273916 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:41.273887    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74b854cf7b-bmgdk"
Sep 07 22:40:41.402145 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:41.402105    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq7qp\" (UniqueName: \"kubernetes.io/projected/67f39360-19f3-4f04-90fc-dc23223b65ea-kube-api-access-bq7qp\") pod \"67f39360-19f3-4f04-90fc-dc23223b65ea\" (UID: \"67f39360-19f3-4f04-90fc-dc23223b65ea\") "
Sep 07 22:40:41.404184 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:41.404156    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f39360-19f3-4f04-90fc-dc23223b65ea-kube-api-access-bq7qp" (OuterVolumeSpecName: "kube-api-access-bq7qp") pod "67f39360-19f3-4f04-90fc-dc23223b65ea" (UID: "67f39360-19f3-4f04-90fc-dc23223b65ea"). InnerVolumeSpecName "kube-api-access-bq7qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:40:41.503102 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:41.503063    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq7qp\" (UniqueName: \"kubernetes.io/projected/67f39360-19f3-4f04-90fc-dc23223b65ea-kube-api-access-bq7qp\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:40:41.993629 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:41.993594    2567 generic.go:356] "Generic (PLEG): container finished" podID="67f39360-19f3-4f04-90fc-dc23223b65ea" containerID="05e50d4a656c9d4619f776ab22b4654ed0a878c188982eebc523d521f57dfed4" exitCode=0
Sep 07 22:40:41.994104 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:41.993638    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74b854cf7b-bmgdk"
Sep 07 22:40:41.994104 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:41.993638    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74b854cf7b-bmgdk" event={"ID":"67f39360-19f3-4f04-90fc-dc23223b65ea","Type":"ContainerDied","Data":"05e50d4a656c9d4619f776ab22b4654ed0a878c188982eebc523d521f57dfed4"}
Sep 07 22:40:41.994104 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:41.993739    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74b854cf7b-bmgdk" event={"ID":"67f39360-19f3-4f04-90fc-dc23223b65ea","Type":"ContainerDied","Data":"11b242134735129a6372ac037a1bdeb496c6b02e75d57113f24670f742ee469c"}
Sep 07 22:40:41.994104 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:41.993758    2567 scope.go:117] "RemoveContainer" containerID="05e50d4a656c9d4619f776ab22b4654ed0a878c188982eebc523d521f57dfed4"
Sep 07 22:40:42.001973 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:42.001954    2567 scope.go:117] "RemoveContainer" containerID="05e50d4a656c9d4619f776ab22b4654ed0a878c188982eebc523d521f57dfed4"
Sep 07 22:40:42.002305 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:40:42.002286    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e50d4a656c9d4619f776ab22b4654ed0a878c188982eebc523d521f57dfed4\": container with ID starting with 05e50d4a656c9d4619f776ab22b4654ed0a878c188982eebc523d521f57dfed4 not found: ID does not exist" containerID="05e50d4a656c9d4619f776ab22b4654ed0a878c188982eebc523d521f57dfed4"
Sep 07 22:40:42.002355 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:42.002314    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e50d4a656c9d4619f776ab22b4654ed0a878c188982eebc523d521f57dfed4"} err="failed to get container status \"05e50d4a656c9d4619f776ab22b4654ed0a878c188982eebc523d521f57dfed4\": rpc error: code = NotFound desc = could not find container \"05e50d4a656c9d4619f776ab22b4654ed0a878c188982eebc523d521f57dfed4\": container with ID starting with 05e50d4a656c9d4619f776ab22b4654ed0a878c188982eebc523d521f57dfed4 not found: ID does not exist"
Sep 07 22:40:42.007015 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:42.006981    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74b854cf7b-bmgdk"]
Sep 07 22:40:42.009673 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:42.009648    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-74b854cf7b-bmgdk"]
Sep 07 22:40:43.465082 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.465044    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67f39360-19f3-4f04-90fc-dc23223b65ea" path="/var/lib/kubelet/pods/67f39360-19f3-4f04-90fc-dc23223b65ea/volumes"
Sep 07 22:40:43.558352 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.558320    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c8b987fd8-ng87k"]
Sep 07 22:40:43.558666 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.558654    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f39360-19f3-4f04-90fc-dc23223b65ea" containerName="authorino"
Sep 07 22:40:43.558711 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.558668    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f39360-19f3-4f04-90fc-dc23223b65ea" containerName="authorino"
Sep 07 22:40:43.558745 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.558734    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f39360-19f3-4f04-90fc-dc23223b65ea" containerName="authorino"
Sep 07 22:40:43.773168 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.773080    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c8b987fd8-ng87k"]
Sep 07 22:40:43.773168 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.773154    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65f4bf68db-w46tm"]
Sep 07 22:40:43.773378 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.773151    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c8b987fd8-ng87k"
Sep 07 22:40:43.776841 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.776809    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 07 22:40:43.776998 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.776885    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 07 22:40:43.776998 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.776961    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9hxct\""
Sep 07 22:40:43.777171 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.777150    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 07 22:40:43.792653 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.792627    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f4bf68db-w46tm"]
Sep 07 22:40:43.792790 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.792722    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f4bf68db-w46tm"
Sep 07 22:40:43.819633 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.819590    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2xt6\" (UniqueName: \"kubernetes.io/projected/e3ce7e38-f199-4407-8e86-f503cca27897-kube-api-access-p2xt6\") pod \"maas-controller-6c8b987fd8-ng87k\" (UID: \"e3ce7e38-f199-4407-8e86-f503cca27897\") " pod="opendatahub/maas-controller-6c8b987fd8-ng87k"
Sep 07 22:40:43.819633 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.819635    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d760100e-6635-4866-b622-3f84c7a52756-webhook-cert\") pod \"maas-controller-65f4bf68db-w46tm\" (UID: \"d760100e-6635-4866-b622-3f84c7a52756\") " pod="opendatahub/maas-controller-65f4bf68db-w46tm"
Sep 07 22:40:43.819850 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.819657    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ce7e38-f199-4407-8e86-f503cca27897-metrics-certs\") pod \"maas-controller-6c8b987fd8-ng87k\" (UID: \"e3ce7e38-f199-4407-8e86-f503cca27897\") " pod="opendatahub/maas-controller-6c8b987fd8-ng87k"
Sep 07 22:40:43.819850 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.819683    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3ce7e38-f199-4407-8e86-f503cca27897-webhook-cert\") pod \"maas-controller-6c8b987fd8-ng87k\" (UID: \"e3ce7e38-f199-4407-8e86-f503cca27897\") " pod="opendatahub/maas-controller-6c8b987fd8-ng87k"
Sep 07 22:40:43.819850 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.819714    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfllk\" (UniqueName: \"kubernetes.io/projected/d760100e-6635-4866-b622-3f84c7a52756-kube-api-access-vfllk\") pod \"maas-controller-65f4bf68db-w46tm\" (UID: \"d760100e-6635-4866-b622-3f84c7a52756\") " pod="opendatahub/maas-controller-65f4bf68db-w46tm"
Sep 07 22:40:43.819850 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.819734    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d760100e-6635-4866-b622-3f84c7a52756-metrics-certs\") pod \"maas-controller-65f4bf68db-w46tm\" (UID: \"d760100e-6635-4866-b622-3f84c7a52756\") " pod="opendatahub/maas-controller-65f4bf68db-w46tm"
Sep 07 22:40:43.920360 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.920325    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2xt6\" (UniqueName: \"kubernetes.io/projected/e3ce7e38-f199-4407-8e86-f503cca27897-kube-api-access-p2xt6\") pod \"maas-controller-6c8b987fd8-ng87k\" (UID: \"e3ce7e38-f199-4407-8e86-f503cca27897\") " pod="opendatahub/maas-controller-6c8b987fd8-ng87k"
Sep 07 22:40:43.920564 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.920378    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d760100e-6635-4866-b622-3f84c7a52756-webhook-cert\") pod \"maas-controller-65f4bf68db-w46tm\" (UID: \"d760100e-6635-4866-b622-3f84c7a52756\") " pod="opendatahub/maas-controller-65f4bf68db-w46tm"
Sep 07 22:40:43.920564 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.920419    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ce7e38-f199-4407-8e86-f503cca27897-metrics-certs\") pod \"maas-controller-6c8b987fd8-ng87k\" (UID: \"e3ce7e38-f199-4407-8e86-f503cca27897\") " pod="opendatahub/maas-controller-6c8b987fd8-ng87k"
Sep 07 22:40:43.920564 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.920447    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3ce7e38-f199-4407-8e86-f503cca27897-webhook-cert\") pod \"maas-controller-6c8b987fd8-ng87k\" (UID: \"e3ce7e38-f199-4407-8e86-f503cca27897\") " pod="opendatahub/maas-controller-6c8b987fd8-ng87k"
Sep 07 22:40:43.920564 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.920482    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfllk\" (UniqueName: \"kubernetes.io/projected/d760100e-6635-4866-b622-3f84c7a52756-kube-api-access-vfllk\") pod \"maas-controller-65f4bf68db-w46tm\" (UID: \"d760100e-6635-4866-b622-3f84c7a52756\") " pod="opendatahub/maas-controller-65f4bf68db-w46tm"
Sep 07 22:40:43.920564 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.920515    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d760100e-6635-4866-b622-3f84c7a52756-metrics-certs\") pod \"maas-controller-65f4bf68db-w46tm\" (UID: \"d760100e-6635-4866-b622-3f84c7a52756\") " pod="opendatahub/maas-controller-65f4bf68db-w46tm"
Sep 07 22:40:43.923048 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.923019    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d760100e-6635-4866-b622-3f84c7a52756-metrics-certs\") pod \"maas-controller-65f4bf68db-w46tm\" (UID: \"d760100e-6635-4866-b622-3f84c7a52756\") " pod="opendatahub/maas-controller-65f4bf68db-w46tm"
Sep 07 22:40:43.923048 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.923039    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d760100e-6635-4866-b622-3f84c7a52756-webhook-cert\") pod \"maas-controller-65f4bf68db-w46tm\" (UID: \"d760100e-6635-4866-b622-3f84c7a52756\") " pod="opendatahub/maas-controller-65f4bf68db-w46tm"
Sep 07 22:40:43.923197 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.923081    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3ce7e38-f199-4407-8e86-f503cca27897-webhook-cert\") pod \"maas-controller-6c8b987fd8-ng87k\" (UID: \"e3ce7e38-f199-4407-8e86-f503cca27897\") " pod="opendatahub/maas-controller-6c8b987fd8-ng87k"
Sep 07 22:40:43.923197 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.923118    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ce7e38-f199-4407-8e86-f503cca27897-metrics-certs\") pod \"maas-controller-6c8b987fd8-ng87k\" (UID: \"e3ce7e38-f199-4407-8e86-f503cca27897\") " pod="opendatahub/maas-controller-6c8b987fd8-ng87k"
Sep 07 22:40:43.927949 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.927924    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2xt6\" (UniqueName: \"kubernetes.io/projected/e3ce7e38-f199-4407-8e86-f503cca27897-kube-api-access-p2xt6\") pod \"maas-controller-6c8b987fd8-ng87k\" (UID: \"e3ce7e38-f199-4407-8e86-f503cca27897\") " pod="opendatahub/maas-controller-6c8b987fd8-ng87k"
Sep 07 22:40:43.928081 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:43.928062    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfllk\" (UniqueName: \"kubernetes.io/projected/d760100e-6635-4866-b622-3f84c7a52756-kube-api-access-vfllk\") pod \"maas-controller-65f4bf68db-w46tm\" (UID: \"d760100e-6635-4866-b622-3f84c7a52756\") " pod="opendatahub/maas-controller-65f4bf68db-w46tm"
Sep 07 22:40:44.083604 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:44.083562    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c8b987fd8-ng87k"
Sep 07 22:40:44.102867 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:44.102828    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f4bf68db-w46tm"
Sep 07 22:40:44.386184 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:44.386157    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c8b987fd8-ng87k"]
Sep 07 22:40:44.388212 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:40:44.388183    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ce7e38_f199_4407_8e86_f503cca27897.slice/crio-cf03f0745f532ae82401dbe865d5daedbbc5f1e325db3825f8f20f2eec7c0969 WatchSource:0}: Error finding container cf03f0745f532ae82401dbe865d5daedbbc5f1e325db3825f8f20f2eec7c0969: Status 404 returned error can't find the container with id cf03f0745f532ae82401dbe865d5daedbbc5f1e325db3825f8f20f2eec7c0969
Sep 07 22:40:44.397632 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:44.396409    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f4bf68db-w46tm"]
Sep 07 22:40:44.400432 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:40:44.400406    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd760100e_6635_4866_b622_3f84c7a52756.slice/crio-ac8a862bc1a995e67db499fdc46666a3298d69384655e0209ddf3428f79df36c WatchSource:0}: Error finding container ac8a862bc1a995e67db499fdc46666a3298d69384655e0209ddf3428f79df36c: Status 404 returned error can't find the container with id ac8a862bc1a995e67db499fdc46666a3298d69384655e0209ddf3428f79df36c
Sep 07 22:40:45.012941 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:45.012897    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c8b987fd8-ng87k" event={"ID":"e3ce7e38-f199-4407-8e86-f503cca27897","Type":"ContainerStarted","Data":"cf03f0745f532ae82401dbe865d5daedbbc5f1e325db3825f8f20f2eec7c0969"}
Sep 07 22:40:45.014526 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:45.014455    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f4bf68db-w46tm" event={"ID":"d760100e-6635-4866-b622-3f84c7a52756","Type":"ContainerStarted","Data":"ac8a862bc1a995e67db499fdc46666a3298d69384655e0209ddf3428f79df36c"}
Sep 07 22:40:48.031725 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:48.031683    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c8b987fd8-ng87k" event={"ID":"e3ce7e38-f199-4407-8e86-f503cca27897","Type":"ContainerStarted","Data":"b18cde405a358f2eafea6efdfd2e202c40b38f56ded2217689fde28cbeddeb07"}
Sep 07 22:40:48.032365 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:48.031758    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c8b987fd8-ng87k"
Sep 07 22:40:48.033713 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:48.033674    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f4bf68db-w46tm" event={"ID":"d760100e-6635-4866-b622-3f84c7a52756","Type":"ContainerStarted","Data":"8c29714df366b3dd1030e979d563999527e0cc66b5698e9bc9f865c1eb16a730"}
Sep 07 22:40:48.033865 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:48.033807    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65f4bf68db-w46tm"
Sep 07 22:40:48.056730 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:48.056663    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c8b987fd8-ng87k" podStartSLOduration=2.448318897 podStartE2EDuration="5.056641256s" totalImagesPullingTime="2.608322359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:40:48.055478554 +0000 UTC m=+531.112311516" watchObservedRunningTime="2026-09-07 22:40:48.056641256 +0000 UTC m=+531.113474219"
Sep 07 22:40:48.076791 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:48.076734    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65f4bf68db-w46tm" podStartSLOduration=2.487971233 podStartE2EDuration="5.076718814s" totalImagesPullingTime="2.588747581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:40:48.075223796 +0000 UTC m=+531.132056759" watchObservedRunningTime="2026-09-07 22:40:48.076718814 +0000 UTC m=+531.133551775"
Sep 07 22:40:59.041852 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:59.041819    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65f4bf68db-w46tm"
Sep 07 22:40:59.042317 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:59.041901    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c8b987fd8-ng87k"
Sep 07 22:40:59.099020 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:59.098987    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c8b987fd8-ng87k"]
Sep 07 22:40:59.099203 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:59.099176    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6c8b987fd8-ng87k" podUID="e3ce7e38-f199-4407-8e86-f503cca27897" containerName="manager" containerID="cri-o://b18cde405a358f2eafea6efdfd2e202c40b38f56ded2217689fde28cbeddeb07" gracePeriod=30
Sep 07 22:40:59.343269 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:59.343225    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c8b987fd8-ng87k"
Sep 07 22:40:59.470265 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:59.470209    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2xt6\" (UniqueName: \"kubernetes.io/projected/e3ce7e38-f199-4407-8e86-f503cca27897-kube-api-access-p2xt6\") pod \"e3ce7e38-f199-4407-8e86-f503cca27897\" (UID: \"e3ce7e38-f199-4407-8e86-f503cca27897\") "
Sep 07 22:40:59.470428 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:59.470288    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ce7e38-f199-4407-8e86-f503cca27897-metrics-certs\") pod \"e3ce7e38-f199-4407-8e86-f503cca27897\" (UID: \"e3ce7e38-f199-4407-8e86-f503cca27897\") "
Sep 07 22:40:59.470428 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:59.470357    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3ce7e38-f199-4407-8e86-f503cca27897-webhook-cert\") pod \"e3ce7e38-f199-4407-8e86-f503cca27897\" (UID: \"e3ce7e38-f199-4407-8e86-f503cca27897\") "
Sep 07 22:40:59.472625 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:59.472584    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ce7e38-f199-4407-8e86-f503cca27897-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e3ce7e38-f199-4407-8e86-f503cca27897" (UID: "e3ce7e38-f199-4407-8e86-f503cca27897"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:40:59.472625 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:59.472614    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ce7e38-f199-4407-8e86-f503cca27897-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e3ce7e38-f199-4407-8e86-f503cca27897" (UID: "e3ce7e38-f199-4407-8e86-f503cca27897"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:40:59.472793 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:59.472640    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ce7e38-f199-4407-8e86-f503cca27897-kube-api-access-p2xt6" (OuterVolumeSpecName: "kube-api-access-p2xt6") pod "e3ce7e38-f199-4407-8e86-f503cca27897" (UID: "e3ce7e38-f199-4407-8e86-f503cca27897"). InnerVolumeSpecName "kube-api-access-p2xt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:40:59.572078 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:59.572033    2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3ce7e38-f199-4407-8e86-f503cca27897-webhook-cert\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:40:59.572078 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:59.572070    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2xt6\" (UniqueName: \"kubernetes.io/projected/e3ce7e38-f199-4407-8e86-f503cca27897-kube-api-access-p2xt6\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:40:59.572078 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:40:59.572084    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ce7e38-f199-4407-8e86-f503cca27897-metrics-certs\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:41:00.092709 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:00.092676    2567 generic.go:356] "Generic (PLEG): container finished" podID="e3ce7e38-f199-4407-8e86-f503cca27897" containerID="b18cde405a358f2eafea6efdfd2e202c40b38f56ded2217689fde28cbeddeb07" exitCode=0
Sep 07 22:41:00.093102 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:00.092726    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c8b987fd8-ng87k"
Sep 07 22:41:00.093102 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:00.092767    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c8b987fd8-ng87k" event={"ID":"e3ce7e38-f199-4407-8e86-f503cca27897","Type":"ContainerDied","Data":"b18cde405a358f2eafea6efdfd2e202c40b38f56ded2217689fde28cbeddeb07"}
Sep 07 22:41:00.093102 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:00.092825    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c8b987fd8-ng87k" event={"ID":"e3ce7e38-f199-4407-8e86-f503cca27897","Type":"ContainerDied","Data":"cf03f0745f532ae82401dbe865d5daedbbc5f1e325db3825f8f20f2eec7c0969"}
Sep 07 22:41:00.093102 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:00.092853    2567 scope.go:117] "RemoveContainer" containerID="b18cde405a358f2eafea6efdfd2e202c40b38f56ded2217689fde28cbeddeb07"
Sep 07 22:41:00.101878 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:00.101765    2567 scope.go:117] "RemoveContainer" containerID="b18cde405a358f2eafea6efdfd2e202c40b38f56ded2217689fde28cbeddeb07"
Sep 07 22:41:00.102186 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:41:00.102108    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18cde405a358f2eafea6efdfd2e202c40b38f56ded2217689fde28cbeddeb07\": container with ID starting with b18cde405a358f2eafea6efdfd2e202c40b38f56ded2217689fde28cbeddeb07 not found: ID does not exist" containerID="b18cde405a358f2eafea6efdfd2e202c40b38f56ded2217689fde28cbeddeb07"
Sep 07 22:41:00.102186 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:00.102136    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18cde405a358f2eafea6efdfd2e202c40b38f56ded2217689fde28cbeddeb07"} err="failed to get container status \"b18cde405a358f2eafea6efdfd2e202c40b38f56ded2217689fde28cbeddeb07\": rpc error: code = NotFound desc = could not find container \"b18cde405a358f2eafea6efdfd2e202c40b38f56ded2217689fde28cbeddeb07\": container with ID starting with b18cde405a358f2eafea6efdfd2e202c40b38f56ded2217689fde28cbeddeb07 not found: ID does not exist"
Sep 07 22:41:00.113050 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:00.113017    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c8b987fd8-ng87k"]
Sep 07 22:41:00.116279 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:00.116226    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6c8b987fd8-ng87k"]
Sep 07 22:41:01.465024 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:01.464987    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3ce7e38-f199-4407-8e86-f503cca27897" path="/var/lib/kubelet/pods/e3ce7e38-f199-4407-8e86-f503cca27897/volumes"
Sep 07 22:41:20.358815 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.358774    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"]
Sep 07 22:41:20.359291 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.359130    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3ce7e38-f199-4407-8e86-f503cca27897" containerName="manager"
Sep 07 22:41:20.359291 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.359144    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3ce7e38-f199-4407-8e86-f503cca27897" containerName="manager"
Sep 07 22:41:20.359291 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.359208    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3ce7e38-f199-4407-8e86-f503cca27897" containerName="manager"
Sep 07 22:41:20.365381 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.365357    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 22:41:20.367583 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.367545    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 07 22:41:20.367583 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.367552    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8mcs4\""
Sep 07 22:41:20.367813 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.367722    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 07 22:41:20.370729 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.370706    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"]
Sep 07 22:41:20.459764 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.459727    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18379880-15f3-418c-b1b8-95a73b92aa7e-openshift-service-ca\") pod \"maas-api-58d7778d67-sgsz4\" (UID: \"18379880-15f3-418c-b1b8-95a73b92aa7e\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 22:41:20.459976 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.459774    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18379880-15f3-418c-b1b8-95a73b92aa7e-metrics-certs\") pod \"maas-api-58d7778d67-sgsz4\" (UID: \"18379880-15f3-418c-b1b8-95a73b92aa7e\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 22:41:20.459976 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.459869    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vwx9\" (UniqueName: \"kubernetes.io/projected/18379880-15f3-418c-b1b8-95a73b92aa7e-kube-api-access-2vwx9\") pod \"maas-api-58d7778d67-sgsz4\" (UID: \"18379880-15f3-418c-b1b8-95a73b92aa7e\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 22:41:20.459976 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.459903    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18379880-15f3-418c-b1b8-95a73b92aa7e-maas-api-tls\") pod \"maas-api-58d7778d67-sgsz4\" (UID: \"18379880-15f3-418c-b1b8-95a73b92aa7e\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 22:41:20.560612 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.560566    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18379880-15f3-418c-b1b8-95a73b92aa7e-openshift-service-ca\") pod \"maas-api-58d7778d67-sgsz4\" (UID: \"18379880-15f3-418c-b1b8-95a73b92aa7e\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 22:41:20.560612 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.560612    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18379880-15f3-418c-b1b8-95a73b92aa7e-metrics-certs\") pod \"maas-api-58d7778d67-sgsz4\" (UID: \"18379880-15f3-418c-b1b8-95a73b92aa7e\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 22:41:20.560914 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.560670    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vwx9\" (UniqueName: \"kubernetes.io/projected/18379880-15f3-418c-b1b8-95a73b92aa7e-kube-api-access-2vwx9\") pod \"maas-api-58d7778d67-sgsz4\" (UID: \"18379880-15f3-418c-b1b8-95a73b92aa7e\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 22:41:20.560914 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.560691    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18379880-15f3-418c-b1b8-95a73b92aa7e-maas-api-tls\") pod \"maas-api-58d7778d67-sgsz4\" (UID: \"18379880-15f3-418c-b1b8-95a73b92aa7e\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 22:41:20.560914 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:41:20.560820    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 07 22:41:20.560914 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:41:20.560897    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18379880-15f3-418c-b1b8-95a73b92aa7e-maas-api-tls podName:18379880-15f3-418c-b1b8-95a73b92aa7e nodeName:}" failed. No retries permitted until 2026-09-07 22:41:21.060874574 +0000 UTC m=+564.117707527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/18379880-15f3-418c-b1b8-95a73b92aa7e-maas-api-tls") pod "maas-api-58d7778d67-sgsz4" (UID: "18379880-15f3-418c-b1b8-95a73b92aa7e") : secret "maas-api-serving-cert" not found
Sep 07 22:41:20.560914 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:41:20.560821    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 07 22:41:20.561149 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:41:20.560973    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18379880-15f3-418c-b1b8-95a73b92aa7e-metrics-certs podName:18379880-15f3-418c-b1b8-95a73b92aa7e nodeName:}" failed. No retries permitted until 2026-09-07 22:41:21.060953744 +0000 UTC m=+564.117786686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18379880-15f3-418c-b1b8-95a73b92aa7e-metrics-certs") pod "maas-api-58d7778d67-sgsz4" (UID: "18379880-15f3-418c-b1b8-95a73b92aa7e") : secret "maas-api-metrics-tls" not found
Sep 07 22:41:20.561296 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.561240    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18379880-15f3-418c-b1b8-95a73b92aa7e-openshift-service-ca\") pod \"maas-api-58d7778d67-sgsz4\" (UID: \"18379880-15f3-418c-b1b8-95a73b92aa7e\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 22:41:20.569547 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.569519    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vwx9\" (UniqueName: \"kubernetes.io/projected/18379880-15f3-418c-b1b8-95a73b92aa7e-kube-api-access-2vwx9\") pod \"maas-api-58d7778d67-sgsz4\" (UID: \"18379880-15f3-418c-b1b8-95a73b92aa7e\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 22:41:20.904450 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.904407    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-6vqtn"]
Sep 07 22:41:20.907933 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.907910    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-6vqtn"
Sep 07 22:41:20.910550 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.910526    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 07 22:41:20.910674 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.910643    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vk8m9\""
Sep 07 22:41:20.915904 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:20.915803    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-6vqtn"]
Sep 07 22:41:21.065169 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:21.065133    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18379880-15f3-418c-b1b8-95a73b92aa7e-maas-api-tls\") pod \"maas-api-58d7778d67-sgsz4\" (UID: \"18379880-15f3-418c-b1b8-95a73b92aa7e\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 22:41:21.065360 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:21.065185    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q9qq\" (UniqueName: \"kubernetes.io/projected/d66e33da-96fe-433d-9dd2-3e5f1ef9f021-kube-api-access-5q9qq\") pod \"payload-processing-5b54b7fbb7-6vqtn\" (UID: \"d66e33da-96fe-433d-9dd2-3e5f1ef9f021\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-6vqtn"
Sep 07 22:41:21.065360 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:21.065280    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18379880-15f3-418c-b1b8-95a73b92aa7e-metrics-certs\") pod \"maas-api-58d7778d67-sgsz4\" (UID: \"18379880-15f3-418c-b1b8-95a73b92aa7e\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 22:41:21.065478 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:21.065356    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d66e33da-96fe-433d-9dd2-3e5f1ef9f021-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-6vqtn\" (UID: \"d66e33da-96fe-433d-9dd2-3e5f1ef9f021\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-6vqtn"
Sep 07 22:41:21.068326 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:21.067760    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18379880-15f3-418c-b1b8-95a73b92aa7e-metrics-certs\") pod \"maas-api-58d7778d67-sgsz4\" (UID: \"18379880-15f3-418c-b1b8-95a73b92aa7e\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 22:41:21.068326 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:21.067797    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18379880-15f3-418c-b1b8-95a73b92aa7e-maas-api-tls\") pod \"maas-api-58d7778d67-sgsz4\" (UID: \"18379880-15f3-418c-b1b8-95a73b92aa7e\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 22:41:21.166878 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:21.166781    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q9qq\" (UniqueName: \"kubernetes.io/projected/d66e33da-96fe-433d-9dd2-3e5f1ef9f021-kube-api-access-5q9qq\") pod \"payload-processing-5b54b7fbb7-6vqtn\" (UID: \"d66e33da-96fe-433d-9dd2-3e5f1ef9f021\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-6vqtn"
Sep 07 22:41:21.166878 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:21.166863    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d66e33da-96fe-433d-9dd2-3e5f1ef9f021-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-6vqtn\" (UID: \"d66e33da-96fe-433d-9dd2-3e5f1ef9f021\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-6vqtn"
Sep 07 22:41:21.167860 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:21.167833    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d66e33da-96fe-433d-9dd2-3e5f1ef9f021-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-6vqtn\" (UID: \"d66e33da-96fe-433d-9dd2-3e5f1ef9f021\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-6vqtn"
Sep 07 22:41:21.174063 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:21.174031    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q9qq\" (UniqueName: \"kubernetes.io/projected/d66e33da-96fe-433d-9dd2-3e5f1ef9f021-kube-api-access-5q9qq\") pod \"payload-processing-5b54b7fbb7-6vqtn\" (UID: \"d66e33da-96fe-433d-9dd2-3e5f1ef9f021\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-6vqtn"
Sep 07 22:41:21.219170 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:21.219131    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-6vqtn"
Sep 07 22:41:21.279359 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:21.278515    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 22:41:21.499355 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:21.499324    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-6vqtn"]
Sep 07 22:41:21.502034 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:41:21.502000    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66e33da_96fe_433d_9dd2_3e5f1ef9f021.slice/crio-569ebd329d6cd79ee3cf8bfa3d618fd62b570c736273d07422926f19af344e42 WatchSource:0}: Error finding container 569ebd329d6cd79ee3cf8bfa3d618fd62b570c736273d07422926f19af344e42: Status 404 returned error can't find the container with id 569ebd329d6cd79ee3cf8bfa3d618fd62b570c736273d07422926f19af344e42
Sep 07 22:41:21.560209 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:21.560170    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"]
Sep 07 22:41:21.564309 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:41:21.564283    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18379880_15f3_418c_b1b8_95a73b92aa7e.slice/crio-9f1adc49f8561a4452daf49bac3ec857e53cc6cc58612bc4fee5646b60be3e9e WatchSource:0}: Error finding container 9f1adc49f8561a4452daf49bac3ec857e53cc6cc58612bc4fee5646b60be3e9e: Status 404 returned error can't find the container with id 9f1adc49f8561a4452daf49bac3ec857e53cc6cc58612bc4fee5646b60be3e9e
Sep 07 22:41:22.202192 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:22.202140    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-6vqtn" event={"ID":"d66e33da-96fe-433d-9dd2-3e5f1ef9f021","Type":"ContainerStarted","Data":"569ebd329d6cd79ee3cf8bfa3d618fd62b570c736273d07422926f19af344e42"}
Sep 07 22:41:22.204494 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:22.204462    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4" event={"ID":"18379880-15f3-418c-b1b8-95a73b92aa7e","Type":"ContainerStarted","Data":"9f1adc49f8561a4452daf49bac3ec857e53cc6cc58612bc4fee5646b60be3e9e"}
Sep 07 22:41:24.216064 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:24.216014    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4" event={"ID":"18379880-15f3-418c-b1b8-95a73b92aa7e","Type":"ContainerStarted","Data":"3c11ab287281ed1cd2bb40be786af10575599aad6073cde2661bd66d0ef12b68"}
Sep 07 22:41:24.216597 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:24.216281    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 22:41:24.217804 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:24.217770    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-6vqtn" event={"ID":"d66e33da-96fe-433d-9dd2-3e5f1ef9f021","Type":"ContainerStarted","Data":"d30387aba3f105df5f43941d993b2b58ec1a1b1c3032ec4faedebe1906d88659"}
Sep 07 22:41:24.217944 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:24.217824    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-6vqtn"
Sep 07 22:41:24.230897 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:24.230820    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4" podStartSLOduration=2.2702751709999998 podStartE2EDuration="4.230800031s" totalImagesPullingTime="1.96052486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:41:24.22981492 +0000 UTC m=+567.286647880" watchObservedRunningTime="2026-09-07 22:41:24.230800031 +0000 UTC m=+567.287632993"
Sep 07 22:41:24.244394 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:24.244338    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-6vqtn" podStartSLOduration=2.215694566 podStartE2EDuration="4.244319675s" totalImagesPullingTime="2.028625109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:41:24.243288752 +0000 UTC m=+567.300121714" watchObservedRunningTime="2026-09-07 22:41:24.244319675 +0000 UTC m=+567.301152704"
Sep 07 22:41:30.228393 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:30.228361    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 22:41:35.225044 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:35.225011    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-6vqtn"
Sep 07 22:41:57.400734 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:57.400706    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/2.log"
Sep 07 22:41:57.401278 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:41:57.400774    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/2.log"
Sep 07 22:42:27.767103 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.767023    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"]
Sep 07 22:42:27.770477 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.770461    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:27.773707 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.773678    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tklfr\""
Sep 07 22:42:27.773865 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.773690    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 07 22:42:27.773865 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.773719    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 07 22:42:27.773994 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.773886    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 07 22:42:27.779077 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.778990    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"]
Sep 07 22:42:27.860734 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.860693    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2187c27-3fc2-4c2d-b874-f1ec7bc72db4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb\" (UID: \"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:27.860734 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.860731    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2187c27-3fc2-4c2d-b874-f1ec7bc72db4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb\" (UID: \"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:27.860940 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.860826    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8brw\" (UniqueName: \"kubernetes.io/projected/b2187c27-3fc2-4c2d-b874-f1ec7bc72db4-kube-api-access-g8brw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb\" (UID: \"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:27.860940 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.860863    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2187c27-3fc2-4c2d-b874-f1ec7bc72db4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb\" (UID: \"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:27.860940 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.860935    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2187c27-3fc2-4c2d-b874-f1ec7bc72db4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb\" (UID: \"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:27.861040 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.860963    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2187c27-3fc2-4c2d-b874-f1ec7bc72db4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb\" (UID: \"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:27.961514 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.961471    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2187c27-3fc2-4c2d-b874-f1ec7bc72db4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb\" (UID: \"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:27.961686 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.961532    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2187c27-3fc2-4c2d-b874-f1ec7bc72db4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb\" (UID: \"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:27.961686 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.961589    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2187c27-3fc2-4c2d-b874-f1ec7bc72db4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb\" (UID: \"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:27.961686 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.961612    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2187c27-3fc2-4c2d-b874-f1ec7bc72db4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb\" (UID: \"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:27.961686 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.961679    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8brw\" (UniqueName: \"kubernetes.io/projected/b2187c27-3fc2-4c2d-b874-f1ec7bc72db4-kube-api-access-g8brw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb\" (UID: \"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:27.961890 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.961712    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2187c27-3fc2-4c2d-b874-f1ec7bc72db4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb\" (UID: \"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:27.962055 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.962031    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2187c27-3fc2-4c2d-b874-f1ec7bc72db4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb\" (UID: \"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:27.962144 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.962082    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2187c27-3fc2-4c2d-b874-f1ec7bc72db4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb\" (UID: \"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:27.962144 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.962121    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2187c27-3fc2-4c2d-b874-f1ec7bc72db4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb\" (UID: \"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:27.963846 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.963828    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2187c27-3fc2-4c2d-b874-f1ec7bc72db4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb\" (UID: \"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:27.964148 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.964130    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2187c27-3fc2-4c2d-b874-f1ec7bc72db4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb\" (UID: \"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:27.970544 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:27.970513    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8brw\" (UniqueName: \"kubernetes.io/projected/b2187c27-3fc2-4c2d-b874-f1ec7bc72db4-kube-api-access-g8brw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb\" (UID: \"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:28.083365 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:28.083333    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:28.355874 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:28.355846    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"]
Sep 07 22:42:28.358004 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:42:28.357975    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2187c27_3fc2_4c2d_b874_f1ec7bc72db4.slice/crio-3b866f49819789a46a768f2f8a9b2981ea4d8ed7ce8d1508e1e2c7fefefcf60f WatchSource:0}: Error finding container 3b866f49819789a46a768f2f8a9b2981ea4d8ed7ce8d1508e1e2c7fefefcf60f: Status 404 returned error can't find the container with id 3b866f49819789a46a768f2f8a9b2981ea4d8ed7ce8d1508e1e2c7fefefcf60f
Sep 07 22:42:28.359484 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:28.359466    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 22:42:28.522580 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:28.522538    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb" event={"ID":"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4","Type":"ContainerStarted","Data":"3b866f49819789a46a768f2f8a9b2981ea4d8ed7ce8d1508e1e2c7fefefcf60f"}
Sep 07 22:42:29.528643 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:29.528600    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb" event={"ID":"b2187c27-3fc2-4c2d-b874-f1ec7bc72db4","Type":"ContainerStarted","Data":"a41d21be665ac70a14ff9585147a176f2304a91452eda870449636ab6099ef34"}
Sep 07 22:42:29.544629 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:29.544575    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb" podStartSLOduration=1.511186207 podStartE2EDuration="2.544559101s" totalImagesPullingTime="1.033372894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:42:29.542676194 +0000 UTC m=+632.599509155" watchObservedRunningTime="2026-09-07 22:42:29.544559101 +0000 UTC m=+632.601392061"
Sep 07 22:42:38.084315 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:38.084270    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:38.084833 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:38.084355    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:38.096988 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:38.096960    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:38.581804 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:38.581773    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb"
Sep 07 22:42:58.376088 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.376044    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"]
Sep 07 22:42:58.379768 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.379744    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.382106 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.382083    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 07 22:42:58.386208 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.386178    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"]
Sep 07 22:42:58.548235 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.548192    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed3d1c60-be53-4d65-8d4d-9828f14b11ff-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7px2s\" (UID: \"ed3d1c60-be53-4d65-8d4d-9828f14b11ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.548235 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.548229    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed3d1c60-be53-4d65-8d4d-9828f14b11ff-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7px2s\" (UID: \"ed3d1c60-be53-4d65-8d4d-9828f14b11ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.548537 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.548280    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed3d1c60-be53-4d65-8d4d-9828f14b11ff-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7px2s\" (UID: \"ed3d1c60-be53-4d65-8d4d-9828f14b11ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.548537 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.548391    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed3d1c60-be53-4d65-8d4d-9828f14b11ff-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7px2s\" (UID: \"ed3d1c60-be53-4d65-8d4d-9828f14b11ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.548537 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.548422    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g29bv\" (UniqueName: \"kubernetes.io/projected/ed3d1c60-be53-4d65-8d4d-9828f14b11ff-kube-api-access-g29bv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7px2s\" (UID: \"ed3d1c60-be53-4d65-8d4d-9828f14b11ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.548537 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.548453    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed3d1c60-be53-4d65-8d4d-9828f14b11ff-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7px2s\" (UID: \"ed3d1c60-be53-4d65-8d4d-9828f14b11ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.649551 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.649454    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed3d1c60-be53-4d65-8d4d-9828f14b11ff-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7px2s\" (UID: \"ed3d1c60-be53-4d65-8d4d-9828f14b11ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.649551 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.649494    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g29bv\" (UniqueName: \"kubernetes.io/projected/ed3d1c60-be53-4d65-8d4d-9828f14b11ff-kube-api-access-g29bv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7px2s\" (UID: \"ed3d1c60-be53-4d65-8d4d-9828f14b11ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.649775 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.649617    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed3d1c60-be53-4d65-8d4d-9828f14b11ff-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7px2s\" (UID: \"ed3d1c60-be53-4d65-8d4d-9828f14b11ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.649775 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.649723    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed3d1c60-be53-4d65-8d4d-9828f14b11ff-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7px2s\" (UID: \"ed3d1c60-be53-4d65-8d4d-9828f14b11ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.649775 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.649752    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed3d1c60-be53-4d65-8d4d-9828f14b11ff-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7px2s\" (UID: \"ed3d1c60-be53-4d65-8d4d-9828f14b11ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.649930 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.649785    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed3d1c60-be53-4d65-8d4d-9828f14b11ff-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7px2s\" (UID: \"ed3d1c60-be53-4d65-8d4d-9828f14b11ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.650148 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.650121    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed3d1c60-be53-4d65-8d4d-9828f14b11ff-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7px2s\" (UID: \"ed3d1c60-be53-4d65-8d4d-9828f14b11ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.650148 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.650138    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed3d1c60-be53-4d65-8d4d-9828f14b11ff-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7px2s\" (UID: \"ed3d1c60-be53-4d65-8d4d-9828f14b11ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.650354 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.650201    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed3d1c60-be53-4d65-8d4d-9828f14b11ff-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7px2s\" (UID: \"ed3d1c60-be53-4d65-8d4d-9828f14b11ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.651987 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.651966    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed3d1c60-be53-4d65-8d4d-9828f14b11ff-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7px2s\" (UID: \"ed3d1c60-be53-4d65-8d4d-9828f14b11ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.652157 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.652139    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed3d1c60-be53-4d65-8d4d-9828f14b11ff-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7px2s\" (UID: \"ed3d1c60-be53-4d65-8d4d-9828f14b11ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.656718 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.656697    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g29bv\" (UniqueName: \"kubernetes.io/projected/ed3d1c60-be53-4d65-8d4d-9828f14b11ff-kube-api-access-g29bv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7px2s\" (UID: \"ed3d1c60-be53-4d65-8d4d-9828f14b11ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.693093 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.693039    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:42:58.963328 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:58.963303    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"]
Sep 07 22:42:58.965462 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:42:58.965432    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d1c60_be53_4d65_8d4d_9828f14b11ff.slice/crio-4c20be85ccd88e19f7af784d56442c3e0c66b7a5b2a9641d7026b8c0618efca1 WatchSource:0}: Error finding container 4c20be85ccd88e19f7af784d56442c3e0c66b7a5b2a9641d7026b8c0618efca1: Status 404 returned error can't find the container with id 4c20be85ccd88e19f7af784d56442c3e0c66b7a5b2a9641d7026b8c0618efca1
Sep 07 22:42:59.672795 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:59.672761    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s" event={"ID":"ed3d1c60-be53-4d65-8d4d-9828f14b11ff","Type":"ContainerStarted","Data":"3ed36b6206210321ed58524719a146f6b7b3411560cfaa170812ac9121dd96fa"}
Sep 07 22:42:59.673330 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:59.672812    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s" event={"ID":"ed3d1c60-be53-4d65-8d4d-9828f14b11ff","Type":"ContainerStarted","Data":"4c20be85ccd88e19f7af784d56442c3e0c66b7a5b2a9641d7026b8c0618efca1"}
Sep 07 22:42:59.689410 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:42:59.689355    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s" podStartSLOduration=1.507641156 podStartE2EDuration="1.68933928s" totalImagesPullingTime="181.698124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:42:59.687757207 +0000 UTC m=+662.744590171" watchObservedRunningTime="2026-09-07 22:42:59.68933928 +0000 UTC m=+662.746172241"
Sep 07 22:43:08.693461 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:43:08.693415    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:43:08.693870 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:43:08.693497    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:43:08.705982 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:43:08.705953    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:43:08.730403 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:43:08.730371    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7px2s"
Sep 07 22:44:42.882666 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:42.882629    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84449f4bcf-njv47"]
Sep 07 22:44:42.886467 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:42.886443    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84449f4bcf-njv47"
Sep 07 22:44:42.893070 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:42.892995    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84449f4bcf-njv47"]
Sep 07 22:44:42.994857 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:42.994813    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/745ea2bb-3e2f-4303-8f3d-35a7a7684cc4-tls-cert\") pod \"authorino-84449f4bcf-njv47\" (UID: \"745ea2bb-3e2f-4303-8f3d-35a7a7684cc4\") " pod="kuadrant-system/authorino-84449f4bcf-njv47"
Sep 07 22:44:42.994857 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:42.994857    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rk57\" (UniqueName: \"kubernetes.io/projected/745ea2bb-3e2f-4303-8f3d-35a7a7684cc4-kube-api-access-5rk57\") pod \"authorino-84449f4bcf-njv47\" (UID: \"745ea2bb-3e2f-4303-8f3d-35a7a7684cc4\") " pod="kuadrant-system/authorino-84449f4bcf-njv47"
Sep 07 22:44:43.096331 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:43.096291    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/745ea2bb-3e2f-4303-8f3d-35a7a7684cc4-tls-cert\") pod \"authorino-84449f4bcf-njv47\" (UID: \"745ea2bb-3e2f-4303-8f3d-35a7a7684cc4\") " pod="kuadrant-system/authorino-84449f4bcf-njv47"
Sep 07 22:44:43.096331 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:43.096334    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rk57\" (UniqueName: \"kubernetes.io/projected/745ea2bb-3e2f-4303-8f3d-35a7a7684cc4-kube-api-access-5rk57\") pod \"authorino-84449f4bcf-njv47\" (UID: \"745ea2bb-3e2f-4303-8f3d-35a7a7684cc4\") " pod="kuadrant-system/authorino-84449f4bcf-njv47"
Sep 07 22:44:43.098850 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:43.098820    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/745ea2bb-3e2f-4303-8f3d-35a7a7684cc4-tls-cert\") pod \"authorino-84449f4bcf-njv47\" (UID: \"745ea2bb-3e2f-4303-8f3d-35a7a7684cc4\") " pod="kuadrant-system/authorino-84449f4bcf-njv47"
Sep 07 22:44:43.104028 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:43.103994    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rk57\" (UniqueName: \"kubernetes.io/projected/745ea2bb-3e2f-4303-8f3d-35a7a7684cc4-kube-api-access-5rk57\") pod \"authorino-84449f4bcf-njv47\" (UID: \"745ea2bb-3e2f-4303-8f3d-35a7a7684cc4\") " pod="kuadrant-system/authorino-84449f4bcf-njv47"
Sep 07 22:44:43.197238 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:43.197142    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84449f4bcf-njv47"
Sep 07 22:44:43.466405 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:43.466382    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84449f4bcf-njv47"]
Sep 07 22:44:43.468057 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:44:43.468031    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod745ea2bb_3e2f_4303_8f3d_35a7a7684cc4.slice/crio-4ea95577b7037cba657116c01a45af260145e700d9c9750152fc3807173d9c16 WatchSource:0}: Error finding container 4ea95577b7037cba657116c01a45af260145e700d9c9750152fc3807173d9c16: Status 404 returned error can't find the container with id 4ea95577b7037cba657116c01a45af260145e700d9c9750152fc3807173d9c16
Sep 07 22:44:44.168525 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:44.168479    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84449f4bcf-njv47" event={"ID":"745ea2bb-3e2f-4303-8f3d-35a7a7684cc4","Type":"ContainerStarted","Data":"4ea95577b7037cba657116c01a45af260145e700d9c9750152fc3807173d9c16"}
Sep 07 22:44:45.178060 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:45.178015    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84449f4bcf-njv47" event={"ID":"745ea2bb-3e2f-4303-8f3d-35a7a7684cc4","Type":"ContainerStarted","Data":"7e15269ea46038d333094b831e8c276c7b3c7f5880c4b501bc4ddedd0a4ffeb5"}
Sep 07 22:44:45.196194 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:45.196138    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-84449f4bcf-njv47" podStartSLOduration=2.13814254 podStartE2EDuration="3.19611925s" totalImagesPullingTime="1.05797671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:44:45.195929208 +0000 UTC m=+768.252762171" watchObservedRunningTime="2026-09-07 22:44:45.19611925 +0000 UTC m=+768.252952210"
Sep 07 22:44:45.225072 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:45.225028    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f846f8c86-jc7fb"]
Sep 07 22:44:45.225429 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:45.225382    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6f846f8c86-jc7fb" podUID="82fb8c01-058c-4b5e-b3c1-baa2002242b2" containerName="authorino" containerID="cri-o://856c789add86ceaf66819c6955eab40813352cc07fe7502c1079e096f713e0a4" gracePeriod=30
Sep 07 22:44:45.479901 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:45.479868    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f846f8c86-jc7fb"
Sep 07 22:44:45.515354 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:45.515312    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/82fb8c01-058c-4b5e-b3c1-baa2002242b2-tls-cert\") pod \"82fb8c01-058c-4b5e-b3c1-baa2002242b2\" (UID: \"82fb8c01-058c-4b5e-b3c1-baa2002242b2\") "
Sep 07 22:44:45.515354 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:45.515358    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtnfj\" (UniqueName: \"kubernetes.io/projected/82fb8c01-058c-4b5e-b3c1-baa2002242b2-kube-api-access-jtnfj\") pod \"82fb8c01-058c-4b5e-b3c1-baa2002242b2\" (UID: \"82fb8c01-058c-4b5e-b3c1-baa2002242b2\") "
Sep 07 22:44:45.517616 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:45.517576    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fb8c01-058c-4b5e-b3c1-baa2002242b2-kube-api-access-jtnfj" (OuterVolumeSpecName: "kube-api-access-jtnfj") pod "82fb8c01-058c-4b5e-b3c1-baa2002242b2" (UID: "82fb8c01-058c-4b5e-b3c1-baa2002242b2"). InnerVolumeSpecName "kube-api-access-jtnfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:44:45.527407 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:45.527374    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82fb8c01-058c-4b5e-b3c1-baa2002242b2-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "82fb8c01-058c-4b5e-b3c1-baa2002242b2" (UID: "82fb8c01-058c-4b5e-b3c1-baa2002242b2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:44:45.616539 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:45.616484    2567 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/82fb8c01-058c-4b5e-b3c1-baa2002242b2-tls-cert\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:44:45.616539 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:45.616538    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtnfj\" (UniqueName: \"kubernetes.io/projected/82fb8c01-058c-4b5e-b3c1-baa2002242b2-kube-api-access-jtnfj\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:44:46.184555 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:46.184520    2567 generic.go:356] "Generic (PLEG): container finished" podID="82fb8c01-058c-4b5e-b3c1-baa2002242b2" containerID="856c789add86ceaf66819c6955eab40813352cc07fe7502c1079e096f713e0a4" exitCode=0
Sep 07 22:44:46.185109 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:46.184561    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f846f8c86-jc7fb"
Sep 07 22:44:46.185109 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:46.184615    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f846f8c86-jc7fb" event={"ID":"82fb8c01-058c-4b5e-b3c1-baa2002242b2","Type":"ContainerDied","Data":"856c789add86ceaf66819c6955eab40813352cc07fe7502c1079e096f713e0a4"}
Sep 07 22:44:46.185109 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:46.184653    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f846f8c86-jc7fb" event={"ID":"82fb8c01-058c-4b5e-b3c1-baa2002242b2","Type":"ContainerDied","Data":"f53735bd028477f84b0b0152f39702a57a059870024f18d18d9deb49a826a578"}
Sep 07 22:44:46.185109 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:46.184691    2567 scope.go:117] "RemoveContainer" containerID="856c789add86ceaf66819c6955eab40813352cc07fe7502c1079e096f713e0a4"
Sep 07 22:44:46.193967 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:46.193948    2567 scope.go:117] "RemoveContainer" containerID="856c789add86ceaf66819c6955eab40813352cc07fe7502c1079e096f713e0a4"
Sep 07 22:44:46.194290 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:44:46.194265    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"856c789add86ceaf66819c6955eab40813352cc07fe7502c1079e096f713e0a4\": container with ID starting with 856c789add86ceaf66819c6955eab40813352cc07fe7502c1079e096f713e0a4 not found: ID does not exist" containerID="856c789add86ceaf66819c6955eab40813352cc07fe7502c1079e096f713e0a4"
Sep 07 22:44:46.194383 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:46.194300    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856c789add86ceaf66819c6955eab40813352cc07fe7502c1079e096f713e0a4"} err="failed to get container status \"856c789add86ceaf66819c6955eab40813352cc07fe7502c1079e096f713e0a4\": rpc error: code = NotFound desc = could not find container \"856c789add86ceaf66819c6955eab40813352cc07fe7502c1079e096f713e0a4\": container with ID starting with 856c789add86ceaf66819c6955eab40813352cc07fe7502c1079e096f713e0a4 not found: ID does not exist"
Sep 07 22:44:46.205749 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:46.205718    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f846f8c86-jc7fb"]
Sep 07 22:44:46.210321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:46.210288    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f846f8c86-jc7fb"]
Sep 07 22:44:47.466264 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:44:47.466222    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82fb8c01-058c-4b5e-b3c1-baa2002242b2" path="/var/lib/kubelet/pods/82fb8c01-058c-4b5e-b3c1-baa2002242b2/volumes"
Sep 07 22:45:07.848066 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:07.848016    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7590-139840930575664"]
Sep 07 22:45:07.848711 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:07.848597    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82fb8c01-058c-4b5e-b3c1-baa2002242b2" containerName="authorino"
Sep 07 22:45:07.848711 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:07.848619    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="82fb8c01-058c-4b5e-b3c1-baa2002242b2" containerName="authorino"
Sep 07 22:45:07.848833 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:07.848731    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="82fb8c01-058c-4b5e-b3c1-baa2002242b2" containerName="authorino"
Sep 07 22:45:07.853220 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:07.853190    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-139840930575664"
Sep 07 22:45:07.855171 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:07.855144    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7590-139840930575664"]
Sep 07 22:45:07.855414 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:07.855394    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b689c\""
Sep 07 22:45:07.923948 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:07.923904    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bv82\" (UniqueName: \"kubernetes.io/projected/22dfa0a8-3fc0-453f-b444-a5e8abb15ee5-kube-api-access-7bv82\") pod \"test-curl-7590-139840930575664\" (UID: \"22dfa0a8-3fc0-453f-b444-a5e8abb15ee5\") " pod="opendatahub/test-curl-7590-139840930575664"
Sep 07 22:45:08.025403 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:08.025365    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bv82\" (UniqueName: \"kubernetes.io/projected/22dfa0a8-3fc0-453f-b444-a5e8abb15ee5-kube-api-access-7bv82\") pod \"test-curl-7590-139840930575664\" (UID: \"22dfa0a8-3fc0-453f-b444-a5e8abb15ee5\") " pod="opendatahub/test-curl-7590-139840930575664"
Sep 07 22:45:08.033062 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:08.033030    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bv82\" (UniqueName: \"kubernetes.io/projected/22dfa0a8-3fc0-453f-b444-a5e8abb15ee5-kube-api-access-7bv82\") pod \"test-curl-7590-139840930575664\" (UID: \"22dfa0a8-3fc0-453f-b444-a5e8abb15ee5\") " pod="opendatahub/test-curl-7590-139840930575664"
Sep 07 22:45:08.164712 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:08.164617    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-139840930575664"
Sep 07 22:45:08.433501 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:08.433424    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7590-139840930575664"]
Sep 07 22:45:08.436416 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:45:08.436378    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22dfa0a8_3fc0_453f_b444_a5e8abb15ee5.slice/crio-50d7d075b0f7878174a608a24b9e5186aac98f726197bf4f818d390eed9480a0 WatchSource:0}: Error finding container 50d7d075b0f7878174a608a24b9e5186aac98f726197bf4f818d390eed9480a0: Status 404 returned error can't find the container with id 50d7d075b0f7878174a608a24b9e5186aac98f726197bf4f818d390eed9480a0
Sep 07 22:45:09.299297 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:09.299171    2567 generic.go:356] "Generic (PLEG): container finished" podID="22dfa0a8-3fc0-453f-b444-a5e8abb15ee5" containerID="dce8cf5292582866b5da3744f7e3ab07fda009f032d753061322853594faea96" exitCode=0
Sep 07 22:45:09.299297 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:09.299213    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7590-139840930575664" event={"ID":"22dfa0a8-3fc0-453f-b444-a5e8abb15ee5","Type":"ContainerDied","Data":"dce8cf5292582866b5da3744f7e3ab07fda009f032d753061322853594faea96"}
Sep 07 22:45:09.299297 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:09.299240    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7590-139840930575664" event={"ID":"22dfa0a8-3fc0-453f-b444-a5e8abb15ee5","Type":"ContainerStarted","Data":"50d7d075b0f7878174a608a24b9e5186aac98f726197bf4f818d390eed9480a0"}
Sep 07 22:45:10.440027 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.440001    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-139840930575664"
Sep 07 22:45:10.504731 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.504695    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7590-139840930575664_22dfa0a8-3fc0-453f-b444-a5e8abb15ee5/test-curl-7590-139840930575664/0.log"
Sep 07 22:45:10.536630 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.536593    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7590-139840930575664"]
Sep 07 22:45:10.548937 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.548902    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bv82\" (UniqueName: \"kubernetes.io/projected/22dfa0a8-3fc0-453f-b444-a5e8abb15ee5-kube-api-access-7bv82\") pod \"22dfa0a8-3fc0-453f-b444-a5e8abb15ee5\" (UID: \"22dfa0a8-3fc0-453f-b444-a5e8abb15ee5\") "
Sep 07 22:45:10.553519 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.551752    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dfa0a8-3fc0-453f-b444-a5e8abb15ee5-kube-api-access-7bv82" (OuterVolumeSpecName: "kube-api-access-7bv82") pod "22dfa0a8-3fc0-453f-b444-a5e8abb15ee5" (UID: "22dfa0a8-3fc0-453f-b444-a5e8abb15ee5"). InnerVolumeSpecName "kube-api-access-7bv82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:45:10.555662 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.555583    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7590-139840930575664"]
Sep 07 22:45:10.650531 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.650487    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bv82\" (UniqueName: \"kubernetes.io/projected/22dfa0a8-3fc0-453f-b444-a5e8abb15ee5-kube-api-access-7bv82\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:45:10.670258 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.670206    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7590-139840930575664"]
Sep 07 22:45:10.670641 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.670626    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22dfa0a8-3fc0-453f-b444-a5e8abb15ee5" containerName="test-curl-7590-139840930575664"
Sep 07 22:45:10.670688 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.670643    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="22dfa0a8-3fc0-453f-b444-a5e8abb15ee5" containerName="test-curl-7590-139840930575664"
Sep 07 22:45:10.670728 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.670705    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="22dfa0a8-3fc0-453f-b444-a5e8abb15ee5" containerName="test-curl-7590-139840930575664"
Sep 07 22:45:10.673705 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.673686    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-139840930575664"
Sep 07 22:45:10.677286 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.677236    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7590-139840930575664"]
Sep 07 22:45:10.751681 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.751638    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlchh\" (UniqueName: \"kubernetes.io/projected/31a5f334-7052-4b5a-b46a-e8aca52750db-kube-api-access-dlchh\") pod \"test-curl-7590-139840930575664\" (UID: \"31a5f334-7052-4b5a-b46a-e8aca52750db\") " pod="opendatahub/test-curl-7590-139840930575664"
Sep 07 22:45:10.852732 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.852684    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlchh\" (UniqueName: \"kubernetes.io/projected/31a5f334-7052-4b5a-b46a-e8aca52750db-kube-api-access-dlchh\") pod \"test-curl-7590-139840930575664\" (UID: \"31a5f334-7052-4b5a-b46a-e8aca52750db\") " pod="opendatahub/test-curl-7590-139840930575664"
Sep 07 22:45:10.864772 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.864737    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlchh\" (UniqueName: \"kubernetes.io/projected/31a5f334-7052-4b5a-b46a-e8aca52750db-kube-api-access-dlchh\") pod \"test-curl-7590-139840930575664\" (UID: \"31a5f334-7052-4b5a-b46a-e8aca52750db\") " pod="opendatahub/test-curl-7590-139840930575664"
Sep 07 22:45:10.875918 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.875886    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"]
Sep 07 22:45:10.879588 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.879557    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"
Sep 07 22:45:10.881675 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.881648    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-51cb36-serving-cert\""
Sep 07 22:45:10.890492 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.890390    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"]
Sep 07 22:45:10.953818 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.953775    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-maas-api-tls\") pod \"maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j\" (UID: \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"
Sep 07 22:45:10.954025 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.953861    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j46p\" (UniqueName: \"kubernetes.io/projected/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-kube-api-access-6j46p\") pod \"maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j\" (UID: \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"
Sep 07 22:45:10.954025 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.953937    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-metrics-certs\") pod \"maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j\" (UID: \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"
Sep 07 22:45:10.954025 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.953997    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-openshift-service-ca\") pod \"maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j\" (UID: \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"
Sep 07 22:45:10.984377 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:10.984340    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-139840930575664"
Sep 07 22:45:11.054831 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:11.054787    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j46p\" (UniqueName: \"kubernetes.io/projected/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-kube-api-access-6j46p\") pod \"maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j\" (UID: \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"
Sep 07 22:45:11.054831 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:11.054835    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-metrics-certs\") pod \"maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j\" (UID: \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"
Sep 07 22:45:11.055110 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:11.055014    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-openshift-service-ca\") pod \"maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j\" (UID: \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"
Sep 07 22:45:11.055110 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:11.055105    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-maas-api-tls\") pod \"maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j\" (UID: \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"
Sep 07 22:45:11.055291 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:45:11.055272    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-serving-cert: secret "maas-api-e2e-inf-a-51cb36-serving-cert" not found
Sep 07 22:45:11.055358 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:45:11.055339    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-maas-api-tls podName:e7cafc2e-4b30-40aa-8b55-343ba65e5c43 nodeName:}" failed. No retries permitted until 2026-09-07 22:45:11.555320904 +0000 UTC m=+794.612153844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-maas-api-tls") pod "maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j" (UID: "e7cafc2e-4b30-40aa-8b55-343ba65e5c43") : secret "maas-api-e2e-inf-a-51cb36-serving-cert" not found
Sep 07 22:45:11.055675 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:11.055653    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-openshift-service-ca\") pod \"maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j\" (UID: \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"
Sep 07 22:45:11.057603 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:11.057584    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-metrics-certs\") pod \"maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j\" (UID: \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"
Sep 07 22:45:11.063366 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:11.063332    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j46p\" (UniqueName: \"kubernetes.io/projected/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-kube-api-access-6j46p\") pod \"maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j\" (UID: \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"
Sep 07 22:45:11.256598 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:11.256569    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7590-139840930575664"]
Sep 07 22:45:11.259173 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:45:11.259145    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31a5f334_7052_4b5a_b46a_e8aca52750db.slice/crio-ed6b892100c10af47ee500b2731ca47f1824cd9b7ffb63d6a60d47d21e2387a9 WatchSource:0}: Error finding container ed6b892100c10af47ee500b2731ca47f1824cd9b7ffb63d6a60d47d21e2387a9: Status 404 returned error can't find the container with id ed6b892100c10af47ee500b2731ca47f1824cd9b7ffb63d6a60d47d21e2387a9
Sep 07 22:45:11.311751 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:11.311713    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7590-139840930575664" event={"ID":"31a5f334-7052-4b5a-b46a-e8aca52750db","Type":"ContainerStarted","Data":"ed6b892100c10af47ee500b2731ca47f1824cd9b7ffb63d6a60d47d21e2387a9"}
Sep 07 22:45:11.313158 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:11.313126    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-139840930575664"
Sep 07 22:45:11.313158 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:11.313154    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50d7d075b0f7878174a608a24b9e5186aac98f726197bf4f818d390eed9480a0"
Sep 07 22:45:11.327224 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:11.327188    2567 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7590-139840930575664" oldPodUID="22dfa0a8-3fc0-453f-b444-a5e8abb15ee5" podUID="31a5f334-7052-4b5a-b46a-e8aca52750db"
Sep 07 22:45:11.466461 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:11.466428    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22dfa0a8-3fc0-453f-b444-a5e8abb15ee5" path="/var/lib/kubelet/pods/22dfa0a8-3fc0-453f-b444-a5e8abb15ee5/volumes"
Sep 07 22:45:11.560073 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:11.559972    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-maas-api-tls\") pod \"maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j\" (UID: \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"
Sep 07 22:45:11.562504 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:11.562479    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-maas-api-tls\") pod \"maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j\" (UID: \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"
Sep 07 22:45:11.795169 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:11.795126    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"
Sep 07 22:45:12.097819 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:12.097792    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"]
Sep 07 22:45:12.100531 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:45:12.100488    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7cafc2e_4b30_40aa_8b55_343ba65e5c43.slice/crio-4bb811566256c2b3e41c70136db1eeafc5366795c64263b408dff968424a8d12 WatchSource:0}: Error finding container 4bb811566256c2b3e41c70136db1eeafc5366795c64263b408dff968424a8d12: Status 404 returned error can't find the container with id 4bb811566256c2b3e41c70136db1eeafc5366795c64263b408dff968424a8d12
Sep 07 22:45:12.319950 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:12.319916    2567 generic.go:356] "Generic (PLEG): container finished" podID="31a5f334-7052-4b5a-b46a-e8aca52750db" containerID="dfc24cddc2e10e20545c54a5f9b5981401cd4292b70e3ff0098cd38e30621ee2" exitCode=0
Sep 07 22:45:12.320128 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:12.319989    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7590-139840930575664" event={"ID":"31a5f334-7052-4b5a-b46a-e8aca52750db","Type":"ContainerDied","Data":"dfc24cddc2e10e20545c54a5f9b5981401cd4292b70e3ff0098cd38e30621ee2"}
Sep 07 22:45:12.321283 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:12.321233    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j" event={"ID":"e7cafc2e-4b30-40aa-8b55-343ba65e5c43","Type":"ContainerStarted","Data":"4bb811566256c2b3e41c70136db1eeafc5366795c64263b408dff968424a8d12"}
Sep 07 22:45:13.327457 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:13.327417    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j" event={"ID":"e7cafc2e-4b30-40aa-8b55-343ba65e5c43","Type":"ContainerStarted","Data":"e818b1124f201c3df4697ad50fd01ef6f2f5b53f903f9b4367f616495544ee95"}
Sep 07 22:45:13.327919 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:13.327674    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"
Sep 07 22:45:13.345935 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:13.345882    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j" podStartSLOduration=2.952129795 podStartE2EDuration="3.345862388s" totalImagesPullingTime="393.732593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:45:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:45:13.344520214 +0000 UTC m=+796.401353161" watchObservedRunningTime="2026-09-07 22:45:13.345862388 +0000 UTC m=+796.402695346"
Sep 07 22:45:13.463829 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:13.463802    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-139840930575664"
Sep 07 22:45:13.482042 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:13.482005    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7590-139840930575664_31a5f334-7052-4b5a-b46a-e8aca52750db/test-curl-7590-139840930575664/0.log"
Sep 07 22:45:13.504093 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:13.504040    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7590-139840930575664"]
Sep 07 22:45:13.507746 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:13.507716    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7590-139840930575664"]
Sep 07 22:45:13.579549 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:13.579456    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlchh\" (UniqueName: \"kubernetes.io/projected/31a5f334-7052-4b5a-b46a-e8aca52750db-kube-api-access-dlchh\") pod \"31a5f334-7052-4b5a-b46a-e8aca52750db\" (UID: \"31a5f334-7052-4b5a-b46a-e8aca52750db\") "
Sep 07 22:45:13.581630 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:13.581603    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a5f334-7052-4b5a-b46a-e8aca52750db-kube-api-access-dlchh" (OuterVolumeSpecName: "kube-api-access-dlchh") pod "31a5f334-7052-4b5a-b46a-e8aca52750db" (UID: "31a5f334-7052-4b5a-b46a-e8aca52750db"). InnerVolumeSpecName "kube-api-access-dlchh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:45:13.681135 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:13.681087    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlchh\" (UniqueName: \"kubernetes.io/projected/31a5f334-7052-4b5a-b46a-e8aca52750db-kube-api-access-dlchh\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:45:14.334342 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:14.334310    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed6b892100c10af47ee500b2731ca47f1824cd9b7ffb63d6a60d47d21e2387a9"
Sep 07 22:45:14.334342 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:14.334329    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-139840930575664"
Sep 07 22:45:14.343876 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:45:14.343845    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7590-139840930575664\" is forbidden: User \"system:node:ip-10-0-138-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-99.ec2.internal' and this object" podUID="31a5f334-7052-4b5a-b46a-e8aca52750db" pod="opendatahub/test-curl-7590-139840930575664"
Sep 07 22:45:15.465332 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:15.465293    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31a5f334-7052-4b5a-b46a-e8aca52750db" path="/var/lib/kubelet/pods/31a5f334-7052-4b5a-b46a-e8aca52750db/volumes"
Sep 07 22:45:18.792929 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:18.792896    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"]
Sep 07 22:45:18.793544 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:18.793388    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31a5f334-7052-4b5a-b46a-e8aca52750db" containerName="test-curl-7590-139840930575664"
Sep 07 22:45:18.793544 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:18.793408    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="31a5f334-7052-4b5a-b46a-e8aca52750db" containerName="test-curl-7590-139840930575664"
Sep 07 22:45:18.793544 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:18.793525    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="31a5f334-7052-4b5a-b46a-e8aca52750db" containerName="test-curl-7590-139840930575664"
Sep 07 22:45:18.797128 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:18.797094    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"
Sep 07 22:45:18.799227 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:18.799208    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-7c3369-serving-cert\""
Sep 07 22:45:18.807029 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:18.806630    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"]
Sep 07 22:45:18.934927 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:18.934888    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d324675-10ff-4300-9187-af1e8d9a2e94-metrics-certs\") pod \"maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w\" (UID: \"1d324675-10ff-4300-9187-af1e8d9a2e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"
Sep 07 22:45:18.935108 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:18.935023    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d324675-10ff-4300-9187-af1e8d9a2e94-maas-api-tls\") pod \"maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w\" (UID: \"1d324675-10ff-4300-9187-af1e8d9a2e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"
Sep 07 22:45:18.935108 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:18.935054    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d324675-10ff-4300-9187-af1e8d9a2e94-openshift-service-ca\") pod \"maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w\" (UID: \"1d324675-10ff-4300-9187-af1e8d9a2e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"
Sep 07 22:45:18.935108 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:18.935081    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cdnn\" (UniqueName: \"kubernetes.io/projected/1d324675-10ff-4300-9187-af1e8d9a2e94-kube-api-access-4cdnn\") pod \"maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w\" (UID: \"1d324675-10ff-4300-9187-af1e8d9a2e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"
Sep 07 22:45:19.035761 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.035721    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d324675-10ff-4300-9187-af1e8d9a2e94-maas-api-tls\") pod \"maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w\" (UID: \"1d324675-10ff-4300-9187-af1e8d9a2e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"
Sep 07 22:45:19.035761 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.035766    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d324675-10ff-4300-9187-af1e8d9a2e94-openshift-service-ca\") pod \"maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w\" (UID: \"1d324675-10ff-4300-9187-af1e8d9a2e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"
Sep 07 22:45:19.036037 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.035796    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cdnn\" (UniqueName: \"kubernetes.io/projected/1d324675-10ff-4300-9187-af1e8d9a2e94-kube-api-access-4cdnn\") pod \"maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w\" (UID: \"1d324675-10ff-4300-9187-af1e8d9a2e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"
Sep 07 22:45:19.036037 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.035870    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d324675-10ff-4300-9187-af1e8d9a2e94-metrics-certs\") pod \"maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w\" (UID: \"1d324675-10ff-4300-9187-af1e8d9a2e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"
Sep 07 22:45:19.036535 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.036498    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d324675-10ff-4300-9187-af1e8d9a2e94-openshift-service-ca\") pod \"maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w\" (UID: \"1d324675-10ff-4300-9187-af1e8d9a2e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"
Sep 07 22:45:19.038300 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.038239    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d324675-10ff-4300-9187-af1e8d9a2e94-metrics-certs\") pod \"maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w\" (UID: \"1d324675-10ff-4300-9187-af1e8d9a2e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"
Sep 07 22:45:19.038446 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.038424    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d324675-10ff-4300-9187-af1e8d9a2e94-maas-api-tls\") pod \"maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w\" (UID: \"1d324675-10ff-4300-9187-af1e8d9a2e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"
Sep 07 22:45:19.044504 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.044444    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cdnn\" (UniqueName: \"kubernetes.io/projected/1d324675-10ff-4300-9187-af1e8d9a2e94-kube-api-access-4cdnn\") pod \"maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w\" (UID: \"1d324675-10ff-4300-9187-af1e8d9a2e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"
Sep 07 22:45:19.111454 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.111415    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"
Sep 07 22:45:19.340469 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.340439    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"
Sep 07 22:45:19.395497 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.395458    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz"]
Sep 07 22:45:19.400485 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.400445    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"]
Sep 07 22:45:19.400632 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.400608    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz"
Sep 07 22:45:19.402871 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:45:19.402844    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d324675_10ff_4300_9187_af1e8d9a2e94.slice/crio-78dc18f6ce3ceec3e57a917ed591ce530486f3eaf9093eb95e456847598fedbd WatchSource:0}: Error finding container 78dc18f6ce3ceec3e57a917ed591ce530486f3eaf9093eb95e456847598fedbd: Status 404 returned error can't find the container with id 78dc18f6ce3ceec3e57a917ed591ce530486f3eaf9093eb95e456847598fedbd
Sep 07 22:45:19.402986 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.402968    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-7c3369-dockercfg-66md9\""
Sep 07 22:45:19.403054 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.402998    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-7c3369\""
Sep 07 22:45:19.407640 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.407573    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz"]
Sep 07 22:45:19.418295 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.418268    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd"]
Sep 07 22:45:19.423289 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.423220    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd"
Sep 07 22:45:19.434398 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.434368    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd"]
Sep 07 22:45:19.541135 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.541092    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2ddp\" (UniqueName: \"kubernetes.io/projected/2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a-kube-api-access-x2ddp\") pod \"payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd\" (UID: \"2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a\") " pod="openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd"
Sep 07 22:45:19.541389 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.541165    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dbadc0-96b7-416f-9abe-db4c5227379d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz\" (UID: \"b3dbadc0-96b7-416f-9abe-db4c5227379d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz"
Sep 07 22:45:19.541389 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.541316    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx2bn\" (UniqueName: \"kubernetes.io/projected/b3dbadc0-96b7-416f-9abe-db4c5227379d-kube-api-access-nx2bn\") pod \"payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz\" (UID: \"b3dbadc0-96b7-416f-9abe-db4c5227379d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz"
Sep 07 22:45:19.541389 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.541373    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a-plugins-config-volume\") pod \"payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd\" (UID: \"2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a\") " pod="openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd"
Sep 07 22:45:19.642528 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.642387    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx2bn\" (UniqueName: \"kubernetes.io/projected/b3dbadc0-96b7-416f-9abe-db4c5227379d-kube-api-access-nx2bn\") pod \"payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz\" (UID: \"b3dbadc0-96b7-416f-9abe-db4c5227379d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz"
Sep 07 22:45:19.642528 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.642443    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a-plugins-config-volume\") pod \"payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd\" (UID: \"2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a\") " pod="openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd"
Sep 07 22:45:19.642528 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.642471    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2ddp\" (UniqueName: \"kubernetes.io/projected/2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a-kube-api-access-x2ddp\") pod \"payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd\" (UID: \"2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a\") " pod="openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd"
Sep 07 22:45:19.642825 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.642537    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dbadc0-96b7-416f-9abe-db4c5227379d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz\" (UID: \"b3dbadc0-96b7-416f-9abe-db4c5227379d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz"
Sep 07 22:45:19.643290 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.643235    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a-plugins-config-volume\") pod \"payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd\" (UID: \"2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a\") " pod="openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd"
Sep 07 22:45:19.643348 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.643324    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dbadc0-96b7-416f-9abe-db4c5227379d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz\" (UID: \"b3dbadc0-96b7-416f-9abe-db4c5227379d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz"
Sep 07 22:45:19.649840 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.649806    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx2bn\" (UniqueName: \"kubernetes.io/projected/b3dbadc0-96b7-416f-9abe-db4c5227379d-kube-api-access-nx2bn\") pod \"payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz\" (UID: \"b3dbadc0-96b7-416f-9abe-db4c5227379d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz"
Sep 07 22:45:19.649840 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.649818    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2ddp\" (UniqueName: \"kubernetes.io/projected/2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a-kube-api-access-x2ddp\") pod \"payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd\" (UID: \"2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a\") " pod="openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd"
Sep 07 22:45:19.724069 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.724031    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz"
Sep 07 22:45:19.737390 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:19.737362    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd"
Sep 07 22:45:20.003014 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:20.002914    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz"]
Sep 07 22:45:20.006368 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:45:20.006338    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3dbadc0_96b7_416f_9abe_db4c5227379d.slice/crio-0cfa5715afb75991d80275926c07ba7832bf3b1661d28e1c7b742100aa49d4aa WatchSource:0}: Error finding container 0cfa5715afb75991d80275926c07ba7832bf3b1661d28e1c7b742100aa49d4aa: Status 404 returned error can't find the container with id 0cfa5715afb75991d80275926c07ba7832bf3b1661d28e1c7b742100aa49d4aa
Sep 07 22:45:20.039746 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:20.039721    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd"]
Sep 07 22:45:20.043119 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:45:20.043081    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2693deeb_20ad_4b52_bd3a_c64fc0b9ff1a.slice/crio-15c9ee27c55c36ec62ceee4e84497c69476c67e824233d0f528e35389ad44e57 WatchSource:0}: Error finding container 15c9ee27c55c36ec62ceee4e84497c69476c67e824233d0f528e35389ad44e57: Status 404 returned error can't find the container with id 15c9ee27c55c36ec62ceee4e84497c69476c67e824233d0f528e35389ad44e57
Sep 07 22:45:20.370729 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:20.370681    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w" event={"ID":"1d324675-10ff-4300-9187-af1e8d9a2e94","Type":"ContainerStarted","Data":"51ec723a1d97ef8f772ae76bb63eb2f79aac3585cce522bc09dbaf2f82edf64e"}
Sep 07 22:45:20.370916 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:20.370747    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w" event={"ID":"1d324675-10ff-4300-9187-af1e8d9a2e94","Type":"ContainerStarted","Data":"78dc18f6ce3ceec3e57a917ed591ce530486f3eaf9093eb95e456847598fedbd"}
Sep 07 22:45:20.370916 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:20.370779    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"
Sep 07 22:45:20.372534 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:20.372493    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd" event={"ID":"2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a","Type":"ContainerStarted","Data":"36fbc132696624e43a6cabcc9bc0dfff38d9838e985255e4e3ce67c08d166412"}
Sep 07 22:45:20.372682 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:20.372555    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd" event={"ID":"2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a","Type":"ContainerStarted","Data":"15c9ee27c55c36ec62ceee4e84497c69476c67e824233d0f528e35389ad44e57"}
Sep 07 22:45:20.372682 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:20.372588    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd"
Sep 07 22:45:20.374228 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:20.374200    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz" event={"ID":"b3dbadc0-96b7-416f-9abe-db4c5227379d","Type":"ContainerStarted","Data":"c1a50a21c507684c17df98286f3646c283c4ea8625fcbb6e72ad0ceb91b2ea72"}
Sep 07 22:45:20.374371 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:20.374261    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz" event={"ID":"b3dbadc0-96b7-416f-9abe-db4c5227379d","Type":"ContainerStarted","Data":"0cfa5715afb75991d80275926c07ba7832bf3b1661d28e1c7b742100aa49d4aa"}
Sep 07 22:45:20.374439 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:20.374372    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz"
Sep 07 22:45:20.388080 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:20.388032    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w" podStartSLOduration=2.032640362 podStartE2EDuration="2.388014861s" totalImagesPullingTime="355.374499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:45:20.385879822 +0000 UTC m=+803.442712799" watchObservedRunningTime="2026-09-07 22:45:20.388014861 +0000 UTC m=+803.444847822"
Sep 07 22:45:20.402693 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:20.402646    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd" podStartSLOduration=1.402629547 podStartE2EDuration="1.402629547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:45:20.401304618 +0000 UTC m=+803.458137583" watchObservedRunningTime="2026-09-07 22:45:20.402629547 +0000 UTC m=+803.459462508"
Sep 07 22:45:20.415652 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:20.415585    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz" podStartSLOduration=1.4155637620000001 podStartE2EDuration="1.415563762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:45:20.415324872 +0000 UTC m=+803.472157834" watchObservedRunningTime="2026-09-07 22:45:20.415563762 +0000 UTC m=+803.472396723"
Sep 07 22:45:21.378657 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:21.378626    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"]
Sep 07 22:45:21.382611 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:21.382572    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"
Sep 07 22:45:21.385072 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:21.385041    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0e3b5bcb-serving-cert\""
Sep 07 22:45:21.404340 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:21.404310    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"]
Sep 07 22:45:21.560853 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:21.560811    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htwmb\" (UniqueName: \"kubernetes.io/projected/339bf899-1130-4f6a-80bb-df6f97e05542-kube-api-access-htwmb\") pod \"maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v\" (UID: \"339bf899-1130-4f6a-80bb-df6f97e05542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"
Sep 07 22:45:21.561040 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:21.560893    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/339bf899-1130-4f6a-80bb-df6f97e05542-maas-api-tls\") pod \"maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v\" (UID: \"339bf899-1130-4f6a-80bb-df6f97e05542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"
Sep 07 22:45:21.561040 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:21.560943    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/339bf899-1130-4f6a-80bb-df6f97e05542-metrics-certs\") pod \"maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v\" (UID: \"339bf899-1130-4f6a-80bb-df6f97e05542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"
Sep 07 22:45:21.561212 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:21.561186    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/339bf899-1130-4f6a-80bb-df6f97e05542-openshift-service-ca\") pod \"maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v\" (UID: \"339bf899-1130-4f6a-80bb-df6f97e05542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"
Sep 07 22:45:21.662628 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:21.662535    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/339bf899-1130-4f6a-80bb-df6f97e05542-openshift-service-ca\") pod \"maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v\" (UID: \"339bf899-1130-4f6a-80bb-df6f97e05542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"
Sep 07 22:45:21.662628 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:21.662612    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htwmb\" (UniqueName: \"kubernetes.io/projected/339bf899-1130-4f6a-80bb-df6f97e05542-kube-api-access-htwmb\") pod \"maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v\" (UID: \"339bf899-1130-4f6a-80bb-df6f97e05542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"
Sep 07 22:45:21.662864 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:21.662649    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/339bf899-1130-4f6a-80bb-df6f97e05542-maas-api-tls\") pod \"maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v\" (UID: \"339bf899-1130-4f6a-80bb-df6f97e05542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"
Sep 07 22:45:21.662864 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:21.662680    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/339bf899-1130-4f6a-80bb-df6f97e05542-metrics-certs\") pod \"maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v\" (UID: \"339bf899-1130-4f6a-80bb-df6f97e05542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"
Sep 07 22:45:21.663223 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:21.663200    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/339bf899-1130-4f6a-80bb-df6f97e05542-openshift-service-ca\") pod \"maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v\" (UID: \"339bf899-1130-4f6a-80bb-df6f97e05542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"
Sep 07 22:45:21.665241 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:21.665213    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/339bf899-1130-4f6a-80bb-df6f97e05542-metrics-certs\") pod \"maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v\" (UID: \"339bf899-1130-4f6a-80bb-df6f97e05542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"
Sep 07 22:45:21.665356 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:21.665337    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/339bf899-1130-4f6a-80bb-df6f97e05542-maas-api-tls\") pod \"maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v\" (UID: \"339bf899-1130-4f6a-80bb-df6f97e05542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"
Sep 07 22:45:21.669932 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:21.669909    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htwmb\" (UniqueName: \"kubernetes.io/projected/339bf899-1130-4f6a-80bb-df6f97e05542-kube-api-access-htwmb\") pod \"maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v\" (UID: \"339bf899-1130-4f6a-80bb-df6f97e05542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"
Sep 07 22:45:21.693706 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:21.693665    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"
Sep 07 22:45:21.969042 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:21.968877    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"]
Sep 07 22:45:21.971337 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:45:21.971303    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339bf899_1130_4f6a_80bb_df6f97e05542.slice/crio-f5b2c297f648d123c29f004a4aa8fe8445fe521ff432f502459aa8881801e185 WatchSource:0}: Error finding container f5b2c297f648d123c29f004a4aa8fe8445fe521ff432f502459aa8881801e185: Status 404 returned error can't find the container with id f5b2c297f648d123c29f004a4aa8fe8445fe521ff432f502459aa8881801e185
Sep 07 22:45:22.389090 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:22.389052    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v" event={"ID":"339bf899-1130-4f6a-80bb-df6f97e05542","Type":"ContainerStarted","Data":"f5b2c297f648d123c29f004a4aa8fe8445fe521ff432f502459aa8881801e185"}
Sep 07 22:45:23.395137 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:23.395102    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v" event={"ID":"339bf899-1130-4f6a-80bb-df6f97e05542","Type":"ContainerStarted","Data":"876c055f0184b64627cab9dd108d65e46d4fdf217ff01d2ab1f58c9372f387bd"}
Sep 07 22:45:23.395562 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:23.395165    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"
Sep 07 22:45:23.411087 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:23.411034    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v" podStartSLOduration=1.960723191 podStartE2EDuration="2.411015222s" totalImagesPullingTime="450.292031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:45:23.409955013 +0000 UTC m=+806.466787976" watchObservedRunningTime="2026-09-07 22:45:23.411015222 +0000 UTC m=+806.467848184"
Sep 07 22:45:26.388117 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:26.388081    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"
Sep 07 22:45:29.405745 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:29.405709    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"
Sep 07 22:45:31.384937 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:31.384903    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz"
Sep 07 22:45:31.385440 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:31.384980    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd"
Sep 07 22:45:38.906174 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:38.906134    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"]
Sep 07 22:45:38.910847 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:38.910819    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"
Sep 07 22:45:38.912789 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:38.912763    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-79b9aa-serving-cert\""
Sep 07 22:45:38.921806 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:38.921777    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"]
Sep 07 22:45:39.033395 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.033351    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f86339cb-5c4a-4e5a-83bc-133659236acf-maas-api-tls\") pod \"maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2\" (UID: \"f86339cb-5c4a-4e5a-83bc-133659236acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"
Sep 07 22:45:39.033567 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.033406    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86339cb-5c4a-4e5a-83bc-133659236acf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2\" (UID: \"f86339cb-5c4a-4e5a-83bc-133659236acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"
Sep 07 22:45:39.033567 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.033469    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cwwb\" (UniqueName: \"kubernetes.io/projected/f86339cb-5c4a-4e5a-83bc-133659236acf-kube-api-access-5cwwb\") pod \"maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2\" (UID: \"f86339cb-5c4a-4e5a-83bc-133659236acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"
Sep 07 22:45:39.033567 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.033519    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f86339cb-5c4a-4e5a-83bc-133659236acf-metrics-certs\") pod \"maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2\" (UID: \"f86339cb-5c4a-4e5a-83bc-133659236acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"
Sep 07 22:45:39.135087 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.135043    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86339cb-5c4a-4e5a-83bc-133659236acf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2\" (UID: \"f86339cb-5c4a-4e5a-83bc-133659236acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"
Sep 07 22:45:39.135313 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.135100    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cwwb\" (UniqueName: \"kubernetes.io/projected/f86339cb-5c4a-4e5a-83bc-133659236acf-kube-api-access-5cwwb\") pod \"maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2\" (UID: \"f86339cb-5c4a-4e5a-83bc-133659236acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"
Sep 07 22:45:39.135313 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.135136    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f86339cb-5c4a-4e5a-83bc-133659236acf-metrics-certs\") pod \"maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2\" (UID: \"f86339cb-5c4a-4e5a-83bc-133659236acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"
Sep 07 22:45:39.135313 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.135289    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f86339cb-5c4a-4e5a-83bc-133659236acf-maas-api-tls\") pod \"maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2\" (UID: \"f86339cb-5c4a-4e5a-83bc-133659236acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"
Sep 07 22:45:39.135801 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.135767    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86339cb-5c4a-4e5a-83bc-133659236acf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2\" (UID: \"f86339cb-5c4a-4e5a-83bc-133659236acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"
Sep 07 22:45:39.137678 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.137654    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f86339cb-5c4a-4e5a-83bc-133659236acf-maas-api-tls\") pod \"maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2\" (UID: \"f86339cb-5c4a-4e5a-83bc-133659236acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"
Sep 07 22:45:39.137794 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.137692    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f86339cb-5c4a-4e5a-83bc-133659236acf-metrics-certs\") pod \"maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2\" (UID: \"f86339cb-5c4a-4e5a-83bc-133659236acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"
Sep 07 22:45:39.142979 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.142947    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cwwb\" (UniqueName: \"kubernetes.io/projected/f86339cb-5c4a-4e5a-83bc-133659236acf-kube-api-access-5cwwb\") pod \"maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2\" (UID: \"f86339cb-5c4a-4e5a-83bc-133659236acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"
Sep 07 22:45:39.226658 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.226565    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"
Sep 07 22:45:39.503039 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.503013    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"]
Sep 07 22:45:39.505600 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:45:39.505570    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86339cb_5c4a_4e5a_83bc_133659236acf.slice/crio-8e4672dfe9e4630d64d1afb77a2adb4468eefca0ea5bcf195fa23f49718247c5 WatchSource:0}: Error finding container 8e4672dfe9e4630d64d1afb77a2adb4468eefca0ea5bcf195fa23f49718247c5: Status 404 returned error can't find the container with id 8e4672dfe9e4630d64d1afb77a2adb4468eefca0ea5bcf195fa23f49718247c5
Sep 07 22:45:39.557291 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.557229    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk"]
Sep 07 22:45:39.562416 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.562387    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk"
Sep 07 22:45:39.564599 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.564572    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-79b9aa-dockercfg-5jtc5\""
Sep 07 22:45:39.564599 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.564587    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-79b9aa\""
Sep 07 22:45:39.573563 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.573530    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk"]
Sep 07 22:45:39.639692 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.639652    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fed04e2-5c55-4330-b97a-a914df8c899d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk\" (UID: \"2fed04e2-5c55-4330-b97a-a914df8c899d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk"
Sep 07 22:45:39.639883 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.639832    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwhdt\" (UniqueName: \"kubernetes.io/projected/2fed04e2-5c55-4330-b97a-a914df8c899d-kube-api-access-qwhdt\") pod \"payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk\" (UID: \"2fed04e2-5c55-4330-b97a-a914df8c899d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk"
Sep 07 22:45:39.740542 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.740497    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwhdt\" (UniqueName: \"kubernetes.io/projected/2fed04e2-5c55-4330-b97a-a914df8c899d-kube-api-access-qwhdt\") pod \"payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk\" (UID: \"2fed04e2-5c55-4330-b97a-a914df8c899d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk"
Sep 07 22:45:39.740746 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.740566    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fed04e2-5c55-4330-b97a-a914df8c899d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk\" (UID: \"2fed04e2-5c55-4330-b97a-a914df8c899d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk"
Sep 07 22:45:39.741219 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.741200    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fed04e2-5c55-4330-b97a-a914df8c899d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk\" (UID: \"2fed04e2-5c55-4330-b97a-a914df8c899d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk"
Sep 07 22:45:39.748567 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.748541    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwhdt\" (UniqueName: \"kubernetes.io/projected/2fed04e2-5c55-4330-b97a-a914df8c899d-kube-api-access-qwhdt\") pod \"payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk\" (UID: \"2fed04e2-5c55-4330-b97a-a914df8c899d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk"
Sep 07 22:45:39.876744 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:39.876719    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk"
Sep 07 22:45:40.156606 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:40.156570    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk"]
Sep 07 22:45:40.159122 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:45:40.159090    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fed04e2_5c55_4330_b97a_a914df8c899d.slice/crio-a9b73b8adca3293d00682d2b41cd3af5ca639bc81a017047c35d416244f8f6aa WatchSource:0}: Error finding container a9b73b8adca3293d00682d2b41cd3af5ca639bc81a017047c35d416244f8f6aa: Status 404 returned error can't find the container with id a9b73b8adca3293d00682d2b41cd3af5ca639bc81a017047c35d416244f8f6aa
Sep 07 22:45:40.491384 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:40.491279    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk" event={"ID":"2fed04e2-5c55-4330-b97a-a914df8c899d","Type":"ContainerStarted","Data":"29979e63f30cfb4d9efb4cd49f076189814a1f9d29e752579af7e7c9613c907a"}
Sep 07 22:45:40.491541 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:40.491479    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk"
Sep 07 22:45:40.491541 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:40.491523    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk" event={"ID":"2fed04e2-5c55-4330-b97a-a914df8c899d","Type":"ContainerStarted","Data":"a9b73b8adca3293d00682d2b41cd3af5ca639bc81a017047c35d416244f8f6aa"}
Sep 07 22:45:40.493020 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:40.492993    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2" event={"ID":"f86339cb-5c4a-4e5a-83bc-133659236acf","Type":"ContainerStarted","Data":"4f0a805d854c738052dc8397be4f87af7b9c95df460a3d88d9b61a526c0a9fed"}
Sep 07 22:45:40.493169 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:40.493048    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2" event={"ID":"f86339cb-5c4a-4e5a-83bc-133659236acf","Type":"ContainerStarted","Data":"8e4672dfe9e4630d64d1afb77a2adb4468eefca0ea5bcf195fa23f49718247c5"}
Sep 07 22:45:40.493169 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:40.493100    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"
Sep 07 22:45:40.508569 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:40.508495    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk" podStartSLOduration=1.5084791929999999 podStartE2EDuration="1.508479193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:45:40.50604658 +0000 UTC m=+823.562879556" watchObservedRunningTime="2026-09-07 22:45:40.508479193 +0000 UTC m=+823.565312207"
Sep 07 22:45:40.522359 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:40.522302    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2" podStartSLOduration=2.207007862 podStartE2EDuration="2.522285526s" totalImagesPullingTime="315.277664ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:45:40.520867748 +0000 UTC m=+823.577700705" watchObservedRunningTime="2026-09-07 22:45:40.522285526 +0000 UTC m=+823.579118486"
Sep 07 22:45:46.503751 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:46.503718    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"
Sep 07 22:45:51.499990 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:45:51.499952    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk"
Sep 07 22:46:01.997801 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:01.997761    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"]
Sep 07 22:46:01.998301 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:01.998019    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v" podUID="339bf899-1130-4f6a-80bb-df6f97e05542" containerName="maas-api" containerID="cri-o://876c055f0184b64627cab9dd108d65e46d4fdf217ff01d2ab1f58c9372f387bd" gracePeriod=30
Sep 07 22:46:02.250148 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.250071    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"
Sep 07 22:46:02.268975 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.268934    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/339bf899-1130-4f6a-80bb-df6f97e05542-maas-api-tls\") pod \"339bf899-1130-4f6a-80bb-df6f97e05542\" (UID: \"339bf899-1130-4f6a-80bb-df6f97e05542\") "
Sep 07 22:46:02.269168 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.269034    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/339bf899-1130-4f6a-80bb-df6f97e05542-metrics-certs\") pod \"339bf899-1130-4f6a-80bb-df6f97e05542\" (UID: \"339bf899-1130-4f6a-80bb-df6f97e05542\") "
Sep 07 22:46:02.269168 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.269092    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/339bf899-1130-4f6a-80bb-df6f97e05542-openshift-service-ca\") pod \"339bf899-1130-4f6a-80bb-df6f97e05542\" (UID: \"339bf899-1130-4f6a-80bb-df6f97e05542\") "
Sep 07 22:46:02.269323 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.269184    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htwmb\" (UniqueName: \"kubernetes.io/projected/339bf899-1130-4f6a-80bb-df6f97e05542-kube-api-access-htwmb\") pod \"339bf899-1130-4f6a-80bb-df6f97e05542\" (UID: \"339bf899-1130-4f6a-80bb-df6f97e05542\") "
Sep 07 22:46:02.269940 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.269672    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339bf899-1130-4f6a-80bb-df6f97e05542-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "339bf899-1130-4f6a-80bb-df6f97e05542" (UID: "339bf899-1130-4f6a-80bb-df6f97e05542"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:46:02.272140 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.272104    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/339bf899-1130-4f6a-80bb-df6f97e05542-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "339bf899-1130-4f6a-80bb-df6f97e05542" (UID: "339bf899-1130-4f6a-80bb-df6f97e05542"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:46:02.272140 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.272127    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/339bf899-1130-4f6a-80bb-df6f97e05542-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "339bf899-1130-4f6a-80bb-df6f97e05542" (UID: "339bf899-1130-4f6a-80bb-df6f97e05542"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:46:02.272359 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.272188    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339bf899-1130-4f6a-80bb-df6f97e05542-kube-api-access-htwmb" (OuterVolumeSpecName: "kube-api-access-htwmb") pod "339bf899-1130-4f6a-80bb-df6f97e05542" (UID: "339bf899-1130-4f6a-80bb-df6f97e05542"). InnerVolumeSpecName "kube-api-access-htwmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:46:02.370522 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.370483    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htwmb\" (UniqueName: \"kubernetes.io/projected/339bf899-1130-4f6a-80bb-df6f97e05542-kube-api-access-htwmb\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:46:02.370522 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.370517    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/339bf899-1130-4f6a-80bb-df6f97e05542-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:46:02.370522 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.370526    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/339bf899-1130-4f6a-80bb-df6f97e05542-metrics-certs\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:46:02.370890 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.370534    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/339bf899-1130-4f6a-80bb-df6f97e05542-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:46:02.615347 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.615312    2567 generic.go:356] "Generic (PLEG): container finished" podID="339bf899-1130-4f6a-80bb-df6f97e05542" containerID="876c055f0184b64627cab9dd108d65e46d4fdf217ff01d2ab1f58c9372f387bd" exitCode=0
Sep 07 22:46:02.615506 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.615355    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v" event={"ID":"339bf899-1130-4f6a-80bb-df6f97e05542","Type":"ContainerDied","Data":"876c055f0184b64627cab9dd108d65e46d4fdf217ff01d2ab1f58c9372f387bd"}
Sep 07 22:46:02.615506 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.615371    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"
Sep 07 22:46:02.615506 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.615401    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v" event={"ID":"339bf899-1130-4f6a-80bb-df6f97e05542","Type":"ContainerDied","Data":"f5b2c297f648d123c29f004a4aa8fe8445fe521ff432f502459aa8881801e185"}
Sep 07 22:46:02.615506 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.615427    2567 scope.go:117] "RemoveContainer" containerID="876c055f0184b64627cab9dd108d65e46d4fdf217ff01d2ab1f58c9372f387bd"
Sep 07 22:46:02.627093 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.627060    2567 scope.go:117] "RemoveContainer" containerID="876c055f0184b64627cab9dd108d65e46d4fdf217ff01d2ab1f58c9372f387bd"
Sep 07 22:46:02.627529 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:46:02.627497    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"876c055f0184b64627cab9dd108d65e46d4fdf217ff01d2ab1f58c9372f387bd\": container with ID starting with 876c055f0184b64627cab9dd108d65e46d4fdf217ff01d2ab1f58c9372f387bd not found: ID does not exist" containerID="876c055f0184b64627cab9dd108d65e46d4fdf217ff01d2ab1f58c9372f387bd"
Sep 07 22:46:02.627627 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.627544    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"876c055f0184b64627cab9dd108d65e46d4fdf217ff01d2ab1f58c9372f387bd"} err="failed to get container status \"876c055f0184b64627cab9dd108d65e46d4fdf217ff01d2ab1f58c9372f387bd\": rpc error: code = NotFound desc = could not find container \"876c055f0184b64627cab9dd108d65e46d4fdf217ff01d2ab1f58c9372f387bd\": container with ID starting with 876c055f0184b64627cab9dd108d65e46d4fdf217ff01d2ab1f58c9372f387bd not found: ID does not exist"
Sep 07 22:46:02.636616 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.636579    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"]
Sep 07 22:46:02.640401 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:02.640368    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e3b5bcb-64865b84b6-mks8v"]
Sep 07 22:46:03.465694 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:03.465648    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="339bf899-1130-4f6a-80bb-df6f97e05542" path="/var/lib/kubelet/pods/339bf899-1130-4f6a-80bb-df6f97e05542/volumes"
Sep 07 22:46:07.312648 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:07.312611    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm"]
Sep 07 22:46:07.313226 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:07.313205    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="339bf899-1130-4f6a-80bb-df6f97e05542" containerName="maas-api"
Sep 07 22:46:07.313347 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:07.313229    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="339bf899-1130-4f6a-80bb-df6f97e05542" containerName="maas-api"
Sep 07 22:46:07.313413 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:07.313387    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="339bf899-1130-4f6a-80bb-df6f97e05542" containerName="maas-api"
Sep 07 22:46:07.317810 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:07.317787    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm"
Sep 07 22:46:07.319881 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:07.319858    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a602f13b-dockercfg-8vjrc\""
Sep 07 22:46:07.319996 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:07.319948    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a602f13b\""
Sep 07 22:46:07.324152 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:07.324128    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm"]
Sep 07 22:46:07.421139 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:07.421097    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4wkh\" (UniqueName: \"kubernetes.io/projected/2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac-kube-api-access-t4wkh\") pod \"payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm\" (UID: \"2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac\") " pod="openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm"
Sep 07 22:46:07.421370 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:07.421262    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm\" (UID: \"2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac\") " pod="openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm"
Sep 07 22:46:07.522821 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:07.522780    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4wkh\" (UniqueName: \"kubernetes.io/projected/2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac-kube-api-access-t4wkh\") pod \"payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm\" (UID: \"2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac\") " pod="openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm"
Sep 07 22:46:07.523127 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:07.523101    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm\" (UID: \"2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac\") " pod="openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm"
Sep 07 22:46:07.523854 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:07.523830    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm\" (UID: \"2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac\") " pod="openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm"
Sep 07 22:46:07.539027 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:07.538998    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4wkh\" (UniqueName: \"kubernetes.io/projected/2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac-kube-api-access-t4wkh\") pod \"payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm\" (UID: \"2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac\") " pod="openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm"
Sep 07 22:46:07.630385 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:07.630350    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm"
Sep 07 22:46:07.915798 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:07.915688    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm"]
Sep 07 22:46:07.918799 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:46:07.918774    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb9b988_57c0_4a5f_ac66_2e8ddbba16ac.slice/crio-8b45b60d3d1a2db81b8477d892e468b8aa361a4a579a2350b8c38b249b019e97 WatchSource:0}: Error finding container 8b45b60d3d1a2db81b8477d892e468b8aa361a4a579a2350b8c38b249b019e97: Status 404 returned error can't find the container with id 8b45b60d3d1a2db81b8477d892e468b8aa361a4a579a2350b8c38b249b019e97
Sep 07 22:46:08.655047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:08.655012    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm" event={"ID":"2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac","Type":"ContainerStarted","Data":"f1c7043c5d29eee42c112cb36ee35480bd22aac9d1ed9265ac3e765215ab60dd"}
Sep 07 22:46:08.655577 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:08.655067    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm" event={"ID":"2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac","Type":"ContainerStarted","Data":"8b45b60d3d1a2db81b8477d892e468b8aa361a4a579a2350b8c38b249b019e97"}
Sep 07 22:46:08.655577 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:08.655091    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm"
Sep 07 22:46:08.673516 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:08.673449    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm" podStartSLOduration=1.673429725 podStartE2EDuration="1.673429725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:46:08.672909134 +0000 UTC m=+851.729742095" watchObservedRunningTime="2026-09-07 22:46:08.673429725 +0000 UTC m=+851.730262686"
Sep 07 22:46:19.668805 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:19.668773    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm"
Sep 07 22:46:37.940812 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:37.940774    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm"]
Sep 07 22:46:37.941237 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:37.941064    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm" podUID="2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac" containerName="payload-processing" containerID="cri-o://f1c7043c5d29eee42c112cb36ee35480bd22aac9d1ed9265ac3e765215ab60dd" gracePeriod=30
Sep 07 22:46:38.217542 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:38.217510    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm"
Sep 07 22:46:38.322468 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:38.322429    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac-plugins-config-volume\") pod \"2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac\" (UID: \"2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac\") "
Sep 07 22:46:38.322616 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:38.322509    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4wkh\" (UniqueName: \"kubernetes.io/projected/2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac-kube-api-access-t4wkh\") pod \"2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac\" (UID: \"2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac\") "
Sep 07 22:46:38.322886 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:38.322863    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac" (UID: "2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:46:38.324667 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:38.324645    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac-kube-api-access-t4wkh" (OuterVolumeSpecName: "kube-api-access-t4wkh") pod "2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac" (UID: "2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac"). InnerVolumeSpecName "kube-api-access-t4wkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:46:38.423497 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:38.423459    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:46:38.423497 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:38.423491    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4wkh\" (UniqueName: \"kubernetes.io/projected/2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac-kube-api-access-t4wkh\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:46:38.817757 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:38.817722    2567 generic.go:356] "Generic (PLEG): container finished" podID="2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac" containerID="f1c7043c5d29eee42c112cb36ee35480bd22aac9d1ed9265ac3e765215ab60dd" exitCode=0
Sep 07 22:46:38.817941 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:38.817765    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm"
Sep 07 22:46:38.817941 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:38.817805    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm" event={"ID":"2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac","Type":"ContainerDied","Data":"f1c7043c5d29eee42c112cb36ee35480bd22aac9d1ed9265ac3e765215ab60dd"}
Sep 07 22:46:38.817941 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:38.817862    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm" event={"ID":"2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac","Type":"ContainerDied","Data":"8b45b60d3d1a2db81b8477d892e468b8aa361a4a579a2350b8c38b249b019e97"}
Sep 07 22:46:38.817941 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:38.817892    2567 scope.go:117] "RemoveContainer" containerID="f1c7043c5d29eee42c112cb36ee35480bd22aac9d1ed9265ac3e765215ab60dd"
Sep 07 22:46:38.828213 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:38.828186    2567 scope.go:117] "RemoveContainer" containerID="f1c7043c5d29eee42c112cb36ee35480bd22aac9d1ed9265ac3e765215ab60dd"
Sep 07 22:46:38.828603 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:46:38.828578    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1c7043c5d29eee42c112cb36ee35480bd22aac9d1ed9265ac3e765215ab60dd\": container with ID starting with f1c7043c5d29eee42c112cb36ee35480bd22aac9d1ed9265ac3e765215ab60dd not found: ID does not exist" containerID="f1c7043c5d29eee42c112cb36ee35480bd22aac9d1ed9265ac3e765215ab60dd"
Sep 07 22:46:38.828668 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:38.828617    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1c7043c5d29eee42c112cb36ee35480bd22aac9d1ed9265ac3e765215ab60dd"} err="failed to get container status \"f1c7043c5d29eee42c112cb36ee35480bd22aac9d1ed9265ac3e765215ab60dd\": rpc error: code = NotFound desc = could not find container \"f1c7043c5d29eee42c112cb36ee35480bd22aac9d1ed9265ac3e765215ab60dd\": container with ID starting with f1c7043c5d29eee42c112cb36ee35480bd22aac9d1ed9265ac3e765215ab60dd not found: ID does not exist"
Sep 07 22:46:38.840656 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:38.840623    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm"]
Sep 07 22:46:38.845984 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:38.845953    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a602f13b-f6d86c49d-pf5rm"]
Sep 07 22:46:39.465895 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:39.465860    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac" path="/var/lib/kubelet/pods/2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac/volumes"
Sep 07 22:46:43.246997 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.246952    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg"]
Sep 07 22:46:43.247805 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.247772    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac" containerName="payload-processing"
Sep 07 22:46:43.247805 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.247802    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac" containerName="payload-processing"
Sep 07 22:46:43.248010 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.247949    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bb9b988-57c0-4a5f-ac66-2e8ddbba16ac" containerName="payload-processing"
Sep 07 22:46:43.252979 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.252952    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg"
Sep 07 22:46:43.255733 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.255706    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-66774a4a\""
Sep 07 22:46:43.255912 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.255786    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-66774a4a-dockercfg-zdf9z\""
Sep 07 22:46:43.261111 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.261076    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg"]
Sep 07 22:46:43.294601 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.294568    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r"]
Sep 07 22:46:43.299324 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.299137    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r"]
Sep 07 22:46:43.299324 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.299292    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r"
Sep 07 22:46:43.382185 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.382149    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzmm6\" (UniqueName: \"kubernetes.io/projected/c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b-kube-api-access-zzmm6\") pod \"payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg\" (UID: \"c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg"
Sep 07 22:46:43.382400 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.382328    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4efc24-70cc-4e20-828c-b5de69d5ab5a-plugins-config-volume\") pod \"payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r\" (UID: \"6f4efc24-70cc-4e20-828c-b5de69d5ab5a\") " pod="openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r"
Sep 07 22:46:43.382481 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.382428    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2nd7\" (UniqueName: \"kubernetes.io/projected/6f4efc24-70cc-4e20-828c-b5de69d5ab5a-kube-api-access-n2nd7\") pod \"payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r\" (UID: \"6f4efc24-70cc-4e20-828c-b5de69d5ab5a\") " pod="openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r"
Sep 07 22:46:43.382481 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.382468    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg\" (UID: \"c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg"
Sep 07 22:46:43.483753 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.483714    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzmm6\" (UniqueName: \"kubernetes.io/projected/c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b-kube-api-access-zzmm6\") pod \"payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg\" (UID: \"c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg"
Sep 07 22:46:43.483958 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.483788    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4efc24-70cc-4e20-828c-b5de69d5ab5a-plugins-config-volume\") pod \"payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r\" (UID: \"6f4efc24-70cc-4e20-828c-b5de69d5ab5a\") " pod="openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r"
Sep 07 22:46:43.483958 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.483823    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2nd7\" (UniqueName: \"kubernetes.io/projected/6f4efc24-70cc-4e20-828c-b5de69d5ab5a-kube-api-access-n2nd7\") pod \"payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r\" (UID: \"6f4efc24-70cc-4e20-828c-b5de69d5ab5a\") " pod="openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r"
Sep 07 22:46:43.483958 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.483844    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg\" (UID: \"c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg"
Sep 07 22:46:43.484579 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.484555    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg\" (UID: \"c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg"
Sep 07 22:46:43.484699 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.484577    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4efc24-70cc-4e20-828c-b5de69d5ab5a-plugins-config-volume\") pod \"payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r\" (UID: \"6f4efc24-70cc-4e20-828c-b5de69d5ab5a\") " pod="openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r"
Sep 07 22:46:43.491940 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.491910    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2nd7\" (UniqueName: \"kubernetes.io/projected/6f4efc24-70cc-4e20-828c-b5de69d5ab5a-kube-api-access-n2nd7\") pod \"payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r\" (UID: \"6f4efc24-70cc-4e20-828c-b5de69d5ab5a\") " pod="openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r"
Sep 07 22:46:43.492069 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.492007    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzmm6\" (UniqueName: \"kubernetes.io/projected/c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b-kube-api-access-zzmm6\") pod \"payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg\" (UID: \"c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg"
Sep 07 22:46:43.605805 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.605759    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg"
Sep 07 22:46:43.615125 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.615101    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r"
Sep 07 22:46:43.878587 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.878489    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg"]
Sep 07 22:46:43.882639 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:46:43.882606    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c8a7c6_22ae_467b_8c4f_85b3ab8b083b.slice/crio-611ca832ee19870c95ef1f7ffe0fb83080683a8e617c2de9af9c27e9695204aa WatchSource:0}: Error finding container 611ca832ee19870c95ef1f7ffe0fb83080683a8e617c2de9af9c27e9695204aa: Status 404 returned error can't find the container with id 611ca832ee19870c95ef1f7ffe0fb83080683a8e617c2de9af9c27e9695204aa
Sep 07 22:46:43.889336 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:43.889314    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r"]
Sep 07 22:46:43.891531 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:46:43.891502    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4efc24_70cc_4e20_828c_b5de69d5ab5a.slice/crio-e1a8375ff36273b47b06c1443a0771b80f176333f92ed2bf480b736333eee065 WatchSource:0}: Error finding container e1a8375ff36273b47b06c1443a0771b80f176333f92ed2bf480b736333eee065: Status 404 returned error can't find the container with id e1a8375ff36273b47b06c1443a0771b80f176333f92ed2bf480b736333eee065
Sep 07 22:46:44.856621 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:44.856579    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r" event={"ID":"6f4efc24-70cc-4e20-828c-b5de69d5ab5a","Type":"ContainerStarted","Data":"f7635fa26abca558a0b55fdbc0b617ff497e276bae42fe8ad23e765367a62f8e"}
Sep 07 22:46:44.857116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:44.856652    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r"
Sep 07 22:46:44.857116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:44.856675    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r" event={"ID":"6f4efc24-70cc-4e20-828c-b5de69d5ab5a","Type":"ContainerStarted","Data":"e1a8375ff36273b47b06c1443a0771b80f176333f92ed2bf480b736333eee065"}
Sep 07 22:46:44.858508 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:44.858479    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg" event={"ID":"c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b","Type":"ContainerStarted","Data":"a00274abe591abb90e2f66c6fce1a284b13c5b3770b6458bee7813d3e380875b"}
Sep 07 22:46:44.858623 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:44.858529    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg" event={"ID":"c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b","Type":"ContainerStarted","Data":"611ca832ee19870c95ef1f7ffe0fb83080683a8e617c2de9af9c27e9695204aa"}
Sep 07 22:46:44.858623 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:44.858563    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg"
Sep 07 22:46:44.874114 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:44.874056    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r" podStartSLOduration=1.874041579 podStartE2EDuration="1.874041579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:46:44.871384581 +0000 UTC m=+887.928217538" watchObservedRunningTime="2026-09-07 22:46:44.874041579 +0000 UTC m=+887.930874539"
Sep 07 22:46:44.886188 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:44.886118    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg" podStartSLOduration=1.886102872 podStartE2EDuration="1.886102872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:46:44.885728745 +0000 UTC m=+887.942561707" watchObservedRunningTime="2026-09-07 22:46:44.886102872 +0000 UTC m=+887.942935835"
Sep 07 22:46:55.868023 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:55.867983    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg"
Sep 07 22:46:55.868602 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:55.868236    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r"
Sep 07 22:46:57.436007 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:57.435969    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/2.log"
Sep 07 22:46:57.436463 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:46:57.436267    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/2.log"
Sep 07 22:47:13.247019 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:13.246980    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r"]
Sep 07 22:47:13.247497 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:13.247266    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r" podUID="6f4efc24-70cc-4e20-828c-b5de69d5ab5a" containerName="payload-processing" containerID="cri-o://f7635fa26abca558a0b55fdbc0b617ff497e276bae42fe8ad23e765367a62f8e" gracePeriod=30
Sep 07 22:47:13.255457 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:13.255427    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg"]
Sep 07 22:47:13.255934 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:13.255733    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg" podUID="c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b" containerName="payload-pre-processing" containerID="cri-o://a00274abe591abb90e2f66c6fce1a284b13c5b3770b6458bee7813d3e380875b" gracePeriod=30
Sep 07 22:47:13.543869 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:13.543842    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r"
Sep 07 22:47:13.546709 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:13.546684    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg"
Sep 07 22:47:13.588594 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:13.588556    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4efc24-70cc-4e20-828c-b5de69d5ab5a-plugins-config-volume\") pod \"6f4efc24-70cc-4e20-828c-b5de69d5ab5a\" (UID: \"6f4efc24-70cc-4e20-828c-b5de69d5ab5a\") "
Sep 07 22:47:13.588594 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:13.588600    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2nd7\" (UniqueName: \"kubernetes.io/projected/6f4efc24-70cc-4e20-828c-b5de69d5ab5a-kube-api-access-n2nd7\") pod \"6f4efc24-70cc-4e20-828c-b5de69d5ab5a\" (UID: \"6f4efc24-70cc-4e20-828c-b5de69d5ab5a\") "
Sep 07 22:47:13.588831 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:13.588674    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzmm6\" (UniqueName: \"kubernetes.io/projected/c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b-kube-api-access-zzmm6\") pod \"c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b\" (UID: \"c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b\") "
Sep 07 22:47:13.588831 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:13.588721    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b-plugins-config-volume\") pod \"c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b\" (UID: \"c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b\") "
Sep 07 22:47:13.589094 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:13.589060    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4efc24-70cc-4e20-828c-b5de69d5ab5a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6f4efc24-70cc-4e20-828c-b5de69d5ab5a" (UID: "6f4efc24-70cc-4e20-828c-b5de69d5ab5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:47:13.589235 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:13.589200    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b" (UID: "c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:47:13.590861 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:13.590834    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b-kube-api-access-zzmm6" (OuterVolumeSpecName: "kube-api-access-zzmm6") pod "c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b" (UID: "c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b"). InnerVolumeSpecName "kube-api-access-zzmm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:47:13.590971 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:13.590874    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4efc24-70cc-4e20-828c-b5de69d5ab5a-kube-api-access-n2nd7" (OuterVolumeSpecName: "kube-api-access-n2nd7") pod "6f4efc24-70cc-4e20-828c-b5de69d5ab5a" (UID: "6f4efc24-70cc-4e20-828c-b5de69d5ab5a"). InnerVolumeSpecName "kube-api-access-n2nd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:47:13.690204 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:13.690162    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4efc24-70cc-4e20-828c-b5de69d5ab5a-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:47:13.690204 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:13.690196    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2nd7\" (UniqueName: \"kubernetes.io/projected/6f4efc24-70cc-4e20-828c-b5de69d5ab5a-kube-api-access-n2nd7\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:47:13.690204 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:13.690209    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzmm6\" (UniqueName: \"kubernetes.io/projected/c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b-kube-api-access-zzmm6\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:47:13.690489 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:13.690218    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:47:14.016702 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:14.016665    2567 generic.go:356] "Generic (PLEG): container finished" podID="6f4efc24-70cc-4e20-828c-b5de69d5ab5a" containerID="f7635fa26abca558a0b55fdbc0b617ff497e276bae42fe8ad23e765367a62f8e" exitCode=0
Sep 07 22:47:14.016909 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:14.016713    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r"
Sep 07 22:47:14.016909 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:14.016762    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r" event={"ID":"6f4efc24-70cc-4e20-828c-b5de69d5ab5a","Type":"ContainerDied","Data":"f7635fa26abca558a0b55fdbc0b617ff497e276bae42fe8ad23e765367a62f8e"}
Sep 07 22:47:14.016909 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:14.016820    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r" event={"ID":"6f4efc24-70cc-4e20-828c-b5de69d5ab5a","Type":"ContainerDied","Data":"e1a8375ff36273b47b06c1443a0771b80f176333f92ed2bf480b736333eee065"}
Sep 07 22:47:14.016909 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:14.016851    2567 scope.go:117] "RemoveContainer" containerID="f7635fa26abca558a0b55fdbc0b617ff497e276bae42fe8ad23e765367a62f8e"
Sep 07 22:47:14.018129 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:14.018103    2567 generic.go:356] "Generic (PLEG): container finished" podID="c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b" containerID="a00274abe591abb90e2f66c6fce1a284b13c5b3770b6458bee7813d3e380875b" exitCode=0
Sep 07 22:47:14.018237 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:14.018145    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg"
Sep 07 22:47:14.018237 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:14.018193    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg" event={"ID":"c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b","Type":"ContainerDied","Data":"a00274abe591abb90e2f66c6fce1a284b13c5b3770b6458bee7813d3e380875b"}
Sep 07 22:47:14.018341 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:14.018260    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg" event={"ID":"c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b","Type":"ContainerDied","Data":"611ca832ee19870c95ef1f7ffe0fb83080683a8e617c2de9af9c27e9695204aa"}
Sep 07 22:47:14.026091 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:14.026066    2567 scope.go:117] "RemoveContainer" containerID="f7635fa26abca558a0b55fdbc0b617ff497e276bae42fe8ad23e765367a62f8e"
Sep 07 22:47:14.026521 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:47:14.026459    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7635fa26abca558a0b55fdbc0b617ff497e276bae42fe8ad23e765367a62f8e\": container with ID starting with f7635fa26abca558a0b55fdbc0b617ff497e276bae42fe8ad23e765367a62f8e not found: ID does not exist" containerID="f7635fa26abca558a0b55fdbc0b617ff497e276bae42fe8ad23e765367a62f8e"
Sep 07 22:47:14.026521 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:14.026497    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7635fa26abca558a0b55fdbc0b617ff497e276bae42fe8ad23e765367a62f8e"} err="failed to get container status \"f7635fa26abca558a0b55fdbc0b617ff497e276bae42fe8ad23e765367a62f8e\": rpc error: code = NotFound desc = could not find container \"f7635fa26abca558a0b55fdbc0b617ff497e276bae42fe8ad23e765367a62f8e\": container with ID starting with f7635fa26abca558a0b55fdbc0b617ff497e276bae42fe8ad23e765367a62f8e not found: ID does not exist"
Sep 07 22:47:14.026672 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:14.026523    2567 scope.go:117] "RemoveContainer" containerID="a00274abe591abb90e2f66c6fce1a284b13c5b3770b6458bee7813d3e380875b"
Sep 07 22:47:14.035271 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:14.035227    2567 scope.go:117] "RemoveContainer" containerID="a00274abe591abb90e2f66c6fce1a284b13c5b3770b6458bee7813d3e380875b"
Sep 07 22:47:14.035731 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:47:14.035710    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a00274abe591abb90e2f66c6fce1a284b13c5b3770b6458bee7813d3e380875b\": container with ID starting with a00274abe591abb90e2f66c6fce1a284b13c5b3770b6458bee7813d3e380875b not found: ID does not exist" containerID="a00274abe591abb90e2f66c6fce1a284b13c5b3770b6458bee7813d3e380875b"
Sep 07 22:47:14.035804 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:14.035742    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00274abe591abb90e2f66c6fce1a284b13c5b3770b6458bee7813d3e380875b"} err="failed to get container status \"a00274abe591abb90e2f66c6fce1a284b13c5b3770b6458bee7813d3e380875b\": rpc error: code = NotFound desc = could not find container \"a00274abe591abb90e2f66c6fce1a284b13c5b3770b6458bee7813d3e380875b\": container with ID starting with a00274abe591abb90e2f66c6fce1a284b13c5b3770b6458bee7813d3e380875b not found: ID does not exist"
Sep 07 22:47:14.040489 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:14.040459    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r"]
Sep 07 22:47:14.043343 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:14.043318    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-66774a4a-677f8d5f87-stz9r"]
Sep 07 22:47:14.052144 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:14.052115    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg"]
Sep 07 22:47:14.065053 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:14.065013    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-66774a4a-7cb6bf8-kvhvg"]
Sep 07 22:47:15.466318 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:15.466281    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f4efc24-70cc-4e20-828c-b5de69d5ab5a" path="/var/lib/kubelet/pods/6f4efc24-70cc-4e20-828c-b5de69d5ab5a/volumes"
Sep 07 22:47:15.466883 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:15.466862    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b" path="/var/lib/kubelet/pods/c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b/volumes"
Sep 07 22:47:19.610821 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:19.610782    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z"]
Sep 07 22:47:19.611338 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:19.611232    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b" containerName="payload-pre-processing"
Sep 07 22:47:19.611338 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:19.611259    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b" containerName="payload-pre-processing"
Sep 07 22:47:19.611338 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:19.611270    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4efc24-70cc-4e20-828c-b5de69d5ab5a" containerName="payload-processing"
Sep 07 22:47:19.611338 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:19.611277    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4efc24-70cc-4e20-828c-b5de69d5ab5a" containerName="payload-processing"
Sep 07 22:47:19.611585 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:19.611344    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f4efc24-70cc-4e20-828c-b5de69d5ab5a" containerName="payload-processing"
Sep 07 22:47:19.611585 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:19.611351    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2c8a7c6-22ae-467b-8c4f-85b3ab8b083b" containerName="payload-pre-processing"
Sep 07 22:47:19.615913 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:19.615891    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z"
Sep 07 22:47:19.618151 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:19.618117    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-01ae3268-dockercfg-xtmpc\""
Sep 07 22:47:19.618303 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:19.618150    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-01ae3268\""
Sep 07 22:47:19.624261 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:19.624216    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z"]
Sep 07 22:47:19.744988 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:19.744942    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqgdq\" (UniqueName: \"kubernetes.io/projected/0d0eaf44-dbd7-4248-bfda-e213807df1e2-kube-api-access-xqgdq\") pod \"payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z\" (UID: \"0d0eaf44-dbd7-4248-bfda-e213807df1e2\") " pod="openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z"
Sep 07 22:47:19.745142 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:19.745062    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0eaf44-dbd7-4248-bfda-e213807df1e2-plugins-config-volume\") pod \"payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z\" (UID: \"0d0eaf44-dbd7-4248-bfda-e213807df1e2\") " pod="openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z"
Sep 07 22:47:19.846401 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:19.846360    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0eaf44-dbd7-4248-bfda-e213807df1e2-plugins-config-volume\") pod \"payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z\" (UID: \"0d0eaf44-dbd7-4248-bfda-e213807df1e2\") " pod="openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z"
Sep 07 22:47:19.846624 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:19.846477    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqgdq\" (UniqueName: \"kubernetes.io/projected/0d0eaf44-dbd7-4248-bfda-e213807df1e2-kube-api-access-xqgdq\") pod \"payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z\" (UID: \"0d0eaf44-dbd7-4248-bfda-e213807df1e2\") " pod="openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z"
Sep 07 22:47:19.847191 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:19.847168    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0eaf44-dbd7-4248-bfda-e213807df1e2-plugins-config-volume\") pod \"payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z\" (UID: \"0d0eaf44-dbd7-4248-bfda-e213807df1e2\") " pod="openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z"
Sep 07 22:47:19.853705 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:19.853663    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqgdq\" (UniqueName: \"kubernetes.io/projected/0d0eaf44-dbd7-4248-bfda-e213807df1e2-kube-api-access-xqgdq\") pod \"payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z\" (UID: \"0d0eaf44-dbd7-4248-bfda-e213807df1e2\") " pod="openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z"
Sep 07 22:47:19.931044 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:19.930949    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z"
Sep 07 22:47:20.206291 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:20.206206    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z"]
Sep 07 22:47:20.208702 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:47:20.208676    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0eaf44_dbd7_4248_bfda_e213807df1e2.slice/crio-2dae5a2b027082ea64e689547c7abbf2921beb8511717900d6d437fcdd26ba45 WatchSource:0}: Error finding container 2dae5a2b027082ea64e689547c7abbf2921beb8511717900d6d437fcdd26ba45: Status 404 returned error can't find the container with id 2dae5a2b027082ea64e689547c7abbf2921beb8511717900d6d437fcdd26ba45
Sep 07 22:47:21.058314 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:21.058274    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z" event={"ID":"0d0eaf44-dbd7-4248-bfda-e213807df1e2","Type":"ContainerStarted","Data":"0a9bb608d73f1b711526d71877b11e7818589ce9895969d642ba9b7a38c7f38d"}
Sep 07 22:47:21.058814 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:21.058346    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z" event={"ID":"0d0eaf44-dbd7-4248-bfda-e213807df1e2","Type":"ContainerStarted","Data":"2dae5a2b027082ea64e689547c7abbf2921beb8511717900d6d437fcdd26ba45"}
Sep 07 22:47:21.058814 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:21.058376    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z"
Sep 07 22:47:21.071736 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:21.071673    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z" podStartSLOduration=2.071652529 podStartE2EDuration="2.071652529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:47:21.071085176 +0000 UTC m=+924.127918138" watchObservedRunningTime="2026-09-07 22:47:21.071652529 +0000 UTC m=+924.128485493"
Sep 07 22:47:25.988680 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:25.988648    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"]
Sep 07 22:47:25.989159 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:25.988861    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w" podUID="1d324675-10ff-4300-9187-af1e8d9a2e94" containerName="maas-api" containerID="cri-o://51ec723a1d97ef8f772ae76bb63eb2f79aac3585cce522bc09dbaf2f82edf64e" gracePeriod=30
Sep 07 22:47:26.037626 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.037589    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd"]
Sep 07 22:47:26.037849 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.037814    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd" podUID="2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a" containerName="payload-processing" containerID="cri-o://36fbc132696624e43a6cabcc9bc0dfff38d9838e985255e4e3ce67c08d166412" gracePeriod=30
Sep 07 22:47:26.046923 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.046886    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz"]
Sep 07 22:47:26.047156 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.047132    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz" podUID="b3dbadc0-96b7-416f-9abe-db4c5227379d" containerName="payload-pre-processing" containerID="cri-o://c1a50a21c507684c17df98286f3646c283c4ea8625fcbb6e72ad0ceb91b2ea72" gracePeriod=30
Sep 07 22:47:26.339983 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.339954    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd"
Sep 07 22:47:26.346061 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.346032    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"
Sep 07 22:47:26.356046 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.356000    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz"
Sep 07 22:47:26.415926 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.415878    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d324675-10ff-4300-9187-af1e8d9a2e94-maas-api-tls\") pod \"1d324675-10ff-4300-9187-af1e8d9a2e94\" (UID: \"1d324675-10ff-4300-9187-af1e8d9a2e94\") "
Sep 07 22:47:26.416122 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.415952    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2ddp\" (UniqueName: \"kubernetes.io/projected/2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a-kube-api-access-x2ddp\") pod \"2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a\" (UID: \"2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a\") "
Sep 07 22:47:26.416122 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.415984    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d324675-10ff-4300-9187-af1e8d9a2e94-metrics-certs\") pod \"1d324675-10ff-4300-9187-af1e8d9a2e94\" (UID: \"1d324675-10ff-4300-9187-af1e8d9a2e94\") "
Sep 07 22:47:26.416122 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.416015    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a-plugins-config-volume\") pod \"2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a\" (UID: \"2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a\") "
Sep 07 22:47:26.416122 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.416066    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cdnn\" (UniqueName: \"kubernetes.io/projected/1d324675-10ff-4300-9187-af1e8d9a2e94-kube-api-access-4cdnn\") pod \"1d324675-10ff-4300-9187-af1e8d9a2e94\" (UID: \"1d324675-10ff-4300-9187-af1e8d9a2e94\") "
Sep 07 22:47:26.416122 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.416100    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dbadc0-96b7-416f-9abe-db4c5227379d-plugins-config-volume\") pod \"b3dbadc0-96b7-416f-9abe-db4c5227379d\" (UID: \"b3dbadc0-96b7-416f-9abe-db4c5227379d\") "
Sep 07 22:47:26.416428 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.416132    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nx2bn\" (UniqueName: \"kubernetes.io/projected/b3dbadc0-96b7-416f-9abe-db4c5227379d-kube-api-access-nx2bn\") pod \"b3dbadc0-96b7-416f-9abe-db4c5227379d\" (UID: \"b3dbadc0-96b7-416f-9abe-db4c5227379d\") "
Sep 07 22:47:26.416428 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.416185    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d324675-10ff-4300-9187-af1e8d9a2e94-openshift-service-ca\") pod \"1d324675-10ff-4300-9187-af1e8d9a2e94\" (UID: \"1d324675-10ff-4300-9187-af1e8d9a2e94\") "
Sep 07 22:47:26.416529 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.416489    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a" (UID: "2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:47:26.416826 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.416638    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3dbadc0-96b7-416f-9abe-db4c5227379d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b3dbadc0-96b7-416f-9abe-db4c5227379d" (UID: "b3dbadc0-96b7-416f-9abe-db4c5227379d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:47:26.416962 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.416910    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d324675-10ff-4300-9187-af1e8d9a2e94-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1d324675-10ff-4300-9187-af1e8d9a2e94" (UID: "1d324675-10ff-4300-9187-af1e8d9a2e94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:47:26.418406 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.418363    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d324675-10ff-4300-9187-af1e8d9a2e94-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1d324675-10ff-4300-9187-af1e8d9a2e94" (UID: "1d324675-10ff-4300-9187-af1e8d9a2e94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:47:26.418406 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.418367    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d324675-10ff-4300-9187-af1e8d9a2e94-kube-api-access-4cdnn" (OuterVolumeSpecName: "kube-api-access-4cdnn") pod "1d324675-10ff-4300-9187-af1e8d9a2e94" (UID: "1d324675-10ff-4300-9187-af1e8d9a2e94"). InnerVolumeSpecName "kube-api-access-4cdnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:47:26.418590 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.418438    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a-kube-api-access-x2ddp" (OuterVolumeSpecName: "kube-api-access-x2ddp") pod "2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a" (UID: "2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a"). InnerVolumeSpecName "kube-api-access-x2ddp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:47:26.418646 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.418608    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3dbadc0-96b7-416f-9abe-db4c5227379d-kube-api-access-nx2bn" (OuterVolumeSpecName: "kube-api-access-nx2bn") pod "b3dbadc0-96b7-416f-9abe-db4c5227379d" (UID: "b3dbadc0-96b7-416f-9abe-db4c5227379d"). InnerVolumeSpecName "kube-api-access-nx2bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:47:26.418710 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.418695    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d324675-10ff-4300-9187-af1e8d9a2e94-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1d324675-10ff-4300-9187-af1e8d9a2e94" (UID: "1d324675-10ff-4300-9187-af1e8d9a2e94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:47:26.517145 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.517040    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d324675-10ff-4300-9187-af1e8d9a2e94-metrics-certs\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:47:26.517145 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.517079    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:47:26.517145 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.517094    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cdnn\" (UniqueName: \"kubernetes.io/projected/1d324675-10ff-4300-9187-af1e8d9a2e94-kube-api-access-4cdnn\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:47:26.517145 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.517106    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dbadc0-96b7-416f-9abe-db4c5227379d-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:47:26.517145 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.517118    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx2bn\" (UniqueName: \"kubernetes.io/projected/b3dbadc0-96b7-416f-9abe-db4c5227379d-kube-api-access-nx2bn\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:47:26.517145 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.517132    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d324675-10ff-4300-9187-af1e8d9a2e94-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:47:26.517145 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.517145    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d324675-10ff-4300-9187-af1e8d9a2e94-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:47:26.517548 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:26.517157    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2ddp\" (UniqueName: \"kubernetes.io/projected/2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a-kube-api-access-x2ddp\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:47:27.093879 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.093842    2567 generic.go:356] "Generic (PLEG): container finished" podID="1d324675-10ff-4300-9187-af1e8d9a2e94" containerID="51ec723a1d97ef8f772ae76bb63eb2f79aac3585cce522bc09dbaf2f82edf64e" exitCode=0
Sep 07 22:47:27.094390 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.093890    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"
Sep 07 22:47:27.094390 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.093921    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w" event={"ID":"1d324675-10ff-4300-9187-af1e8d9a2e94","Type":"ContainerDied","Data":"51ec723a1d97ef8f772ae76bb63eb2f79aac3585cce522bc09dbaf2f82edf64e"}
Sep 07 22:47:27.094390 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.093967    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w" event={"ID":"1d324675-10ff-4300-9187-af1e8d9a2e94","Type":"ContainerDied","Data":"78dc18f6ce3ceec3e57a917ed591ce530486f3eaf9093eb95e456847598fedbd"}
Sep 07 22:47:27.094390 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.093991    2567 scope.go:117] "RemoveContainer" containerID="51ec723a1d97ef8f772ae76bb63eb2f79aac3585cce522bc09dbaf2f82edf64e"
Sep 07 22:47:27.095563 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.095494    2567 generic.go:356] "Generic (PLEG): container finished" podID="2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a" containerID="36fbc132696624e43a6cabcc9bc0dfff38d9838e985255e4e3ce67c08d166412" exitCode=0
Sep 07 22:47:27.095563 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.095533    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd" event={"ID":"2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a","Type":"ContainerDied","Data":"36fbc132696624e43a6cabcc9bc0dfff38d9838e985255e4e3ce67c08d166412"}
Sep 07 22:47:27.095563 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.095542    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd"
Sep 07 22:47:27.095730 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.095578    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd" event={"ID":"2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a","Type":"ContainerDied","Data":"15c9ee27c55c36ec62ceee4e84497c69476c67e824233d0f528e35389ad44e57"}
Sep 07 22:47:27.097163 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.097137    2567 generic.go:356] "Generic (PLEG): container finished" podID="b3dbadc0-96b7-416f-9abe-db4c5227379d" containerID="c1a50a21c507684c17df98286f3646c283c4ea8625fcbb6e72ad0ceb91b2ea72" exitCode=0
Sep 07 22:47:27.097373 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.097169    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz" event={"ID":"b3dbadc0-96b7-416f-9abe-db4c5227379d","Type":"ContainerDied","Data":"c1a50a21c507684c17df98286f3646c283c4ea8625fcbb6e72ad0ceb91b2ea72"}
Sep 07 22:47:27.097373 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.097233    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz" event={"ID":"b3dbadc0-96b7-416f-9abe-db4c5227379d","Type":"ContainerDied","Data":"0cfa5715afb75991d80275926c07ba7832bf3b1661d28e1c7b742100aa49d4aa"}
Sep 07 22:47:27.097373 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.097182    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz"
Sep 07 22:47:27.105229 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.105208    2567 scope.go:117] "RemoveContainer" containerID="51ec723a1d97ef8f772ae76bb63eb2f79aac3585cce522bc09dbaf2f82edf64e"
Sep 07 22:47:27.105609 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:47:27.105589    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ec723a1d97ef8f772ae76bb63eb2f79aac3585cce522bc09dbaf2f82edf64e\": container with ID starting with 51ec723a1d97ef8f772ae76bb63eb2f79aac3585cce522bc09dbaf2f82edf64e not found: ID does not exist" containerID="51ec723a1d97ef8f772ae76bb63eb2f79aac3585cce522bc09dbaf2f82edf64e"
Sep 07 22:47:27.105700 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.105617    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ec723a1d97ef8f772ae76bb63eb2f79aac3585cce522bc09dbaf2f82edf64e"} err="failed to get container status \"51ec723a1d97ef8f772ae76bb63eb2f79aac3585cce522bc09dbaf2f82edf64e\": rpc error: code = NotFound desc = could not find container \"51ec723a1d97ef8f772ae76bb63eb2f79aac3585cce522bc09dbaf2f82edf64e\": container with ID starting with 51ec723a1d97ef8f772ae76bb63eb2f79aac3585cce522bc09dbaf2f82edf64e not found: ID does not exist"
Sep 07 22:47:27.105700 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.105634    2567 scope.go:117] "RemoveContainer" containerID="36fbc132696624e43a6cabcc9bc0dfff38d9838e985255e4e3ce67c08d166412"
Sep 07 22:47:27.114357 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.114337    2567 scope.go:117] "RemoveContainer" containerID="36fbc132696624e43a6cabcc9bc0dfff38d9838e985255e4e3ce67c08d166412"
Sep 07 22:47:27.114657 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:47:27.114635    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36fbc132696624e43a6cabcc9bc0dfff38d9838e985255e4e3ce67c08d166412\": container with ID starting with 36fbc132696624e43a6cabcc9bc0dfff38d9838e985255e4e3ce67c08d166412 not found: ID does not exist" containerID="36fbc132696624e43a6cabcc9bc0dfff38d9838e985255e4e3ce67c08d166412"
Sep 07 22:47:27.114703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.114666    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36fbc132696624e43a6cabcc9bc0dfff38d9838e985255e4e3ce67c08d166412"} err="failed to get container status \"36fbc132696624e43a6cabcc9bc0dfff38d9838e985255e4e3ce67c08d166412\": rpc error: code = NotFound desc = could not find container \"36fbc132696624e43a6cabcc9bc0dfff38d9838e985255e4e3ce67c08d166412\": container with ID starting with 36fbc132696624e43a6cabcc9bc0dfff38d9838e985255e4e3ce67c08d166412 not found: ID does not exist"
Sep 07 22:47:27.114703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.114684    2567 scope.go:117] "RemoveContainer" containerID="c1a50a21c507684c17df98286f3646c283c4ea8625fcbb6e72ad0ceb91b2ea72"
Sep 07 22:47:27.119863 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.119834    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"]
Sep 07 22:47:27.123223 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.123203    2567 scope.go:117] "RemoveContainer" containerID="c1a50a21c507684c17df98286f3646c283c4ea8625fcbb6e72ad0ceb91b2ea72"
Sep 07 22:47:27.123620 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:47:27.123596    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1a50a21c507684c17df98286f3646c283c4ea8625fcbb6e72ad0ceb91b2ea72\": container with ID starting with c1a50a21c507684c17df98286f3646c283c4ea8625fcbb6e72ad0ceb91b2ea72 not found: ID does not exist" containerID="c1a50a21c507684c17df98286f3646c283c4ea8625fcbb6e72ad0ceb91b2ea72"
Sep 07 22:47:27.123681 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.123633    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1a50a21c507684c17df98286f3646c283c4ea8625fcbb6e72ad0ceb91b2ea72"} err="failed to get container status \"c1a50a21c507684c17df98286f3646c283c4ea8625fcbb6e72ad0ceb91b2ea72\": rpc error: code = NotFound desc = could not find container \"c1a50a21c507684c17df98286f3646c283c4ea8625fcbb6e72ad0ceb91b2ea72\": container with ID starting with c1a50a21c507684c17df98286f3646c283c4ea8625fcbb6e72ad0ceb91b2ea72 not found: ID does not exist"
Sep 07 22:47:27.123899 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.123884    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7c3369-5b5cfd4fb-wfg8w"]
Sep 07 22:47:27.144240 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.144201    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz"]
Sep 07 22:47:27.161375 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.161341    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7c3369-7b577898f8-6dxlz"]
Sep 07 22:47:27.182353 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.182319    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd"]
Sep 07 22:47:27.199953 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.199920    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7c3369-5c5c49dd79-wl9gd"]
Sep 07 22:47:27.466482 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.466449    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d324675-10ff-4300-9187-af1e8d9a2e94" path="/var/lib/kubelet/pods/1d324675-10ff-4300-9187-af1e8d9a2e94/volumes"
Sep 07 22:47:27.466841 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.466828    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a" path="/var/lib/kubelet/pods/2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a/volumes"
Sep 07 22:47:27.467144 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:27.467133    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3dbadc0-96b7-416f-9abe-db4c5227379d" path="/var/lib/kubelet/pods/b3dbadc0-96b7-416f-9abe-db4c5227379d/volumes"
Sep 07 22:47:32.066113 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:32.066081    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z"
Sep 07 22:47:49.445413 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:49.445364    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z"]
Sep 07 22:47:49.445893 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:49.445596    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z" podUID="0d0eaf44-dbd7-4248-bfda-e213807df1e2" containerName="payload-processing" containerID="cri-o://0a9bb608d73f1b711526d71877b11e7818589ce9895969d642ba9b7a38c7f38d" gracePeriod=30
Sep 07 22:47:49.696601 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:49.696527    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z"
Sep 07 22:47:49.739353 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:49.739318    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqgdq\" (UniqueName: \"kubernetes.io/projected/0d0eaf44-dbd7-4248-bfda-e213807df1e2-kube-api-access-xqgdq\") pod \"0d0eaf44-dbd7-4248-bfda-e213807df1e2\" (UID: \"0d0eaf44-dbd7-4248-bfda-e213807df1e2\") "
Sep 07 22:47:49.739525 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:49.739425    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0eaf44-dbd7-4248-bfda-e213807df1e2-plugins-config-volume\") pod \"0d0eaf44-dbd7-4248-bfda-e213807df1e2\" (UID: \"0d0eaf44-dbd7-4248-bfda-e213807df1e2\") "
Sep 07 22:47:49.739799 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:49.739767    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d0eaf44-dbd7-4248-bfda-e213807df1e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d0eaf44-dbd7-4248-bfda-e213807df1e2" (UID: "0d0eaf44-dbd7-4248-bfda-e213807df1e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:47:49.741564 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:49.741529    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0eaf44-dbd7-4248-bfda-e213807df1e2-kube-api-access-xqgdq" (OuterVolumeSpecName: "kube-api-access-xqgdq") pod "0d0eaf44-dbd7-4248-bfda-e213807df1e2" (UID: "0d0eaf44-dbd7-4248-bfda-e213807df1e2"). InnerVolumeSpecName "kube-api-access-xqgdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:47:49.841732 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:49.841682    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0eaf44-dbd7-4248-bfda-e213807df1e2-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:47:49.841732 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:49.841728    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqgdq\" (UniqueName: \"kubernetes.io/projected/0d0eaf44-dbd7-4248-bfda-e213807df1e2-kube-api-access-xqgdq\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:47:50.219753 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:50.219717    2567 generic.go:356] "Generic (PLEG): container finished" podID="0d0eaf44-dbd7-4248-bfda-e213807df1e2" containerID="0a9bb608d73f1b711526d71877b11e7818589ce9895969d642ba9b7a38c7f38d" exitCode=0
Sep 07 22:47:50.219926 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:50.219774    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z"
Sep 07 22:47:50.219926 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:50.219811    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z" event={"ID":"0d0eaf44-dbd7-4248-bfda-e213807df1e2","Type":"ContainerDied","Data":"0a9bb608d73f1b711526d71877b11e7818589ce9895969d642ba9b7a38c7f38d"}
Sep 07 22:47:50.219926 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:50.219868    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z" event={"ID":"0d0eaf44-dbd7-4248-bfda-e213807df1e2","Type":"ContainerDied","Data":"2dae5a2b027082ea64e689547c7abbf2921beb8511717900d6d437fcdd26ba45"}
Sep 07 22:47:50.219926 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:50.219891    2567 scope.go:117] "RemoveContainer" containerID="0a9bb608d73f1b711526d71877b11e7818589ce9895969d642ba9b7a38c7f38d"
Sep 07 22:47:50.229798 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:50.229778    2567 scope.go:117] "RemoveContainer" containerID="0a9bb608d73f1b711526d71877b11e7818589ce9895969d642ba9b7a38c7f38d"
Sep 07 22:47:50.230095 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:47:50.230075    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9bb608d73f1b711526d71877b11e7818589ce9895969d642ba9b7a38c7f38d\": container with ID starting with 0a9bb608d73f1b711526d71877b11e7818589ce9895969d642ba9b7a38c7f38d not found: ID does not exist" containerID="0a9bb608d73f1b711526d71877b11e7818589ce9895969d642ba9b7a38c7f38d"
Sep 07 22:47:50.230150 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:50.230105    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9bb608d73f1b711526d71877b11e7818589ce9895969d642ba9b7a38c7f38d"} err="failed to get container status \"0a9bb608d73f1b711526d71877b11e7818589ce9895969d642ba9b7a38c7f38d\": rpc error: code = NotFound desc = could not find container \"0a9bb608d73f1b711526d71877b11e7818589ce9895969d642ba9b7a38c7f38d\": container with ID starting with 0a9bb608d73f1b711526d71877b11e7818589ce9895969d642ba9b7a38c7f38d not found: ID does not exist"
Sep 07 22:47:50.240776 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:50.240748    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z"]
Sep 07 22:47:50.244116 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:50.244092    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-01ae3268-54bfb6689c-kc92z"]
Sep 07 22:47:51.465435 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:51.465405    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d0eaf44-dbd7-4248-bfda-e213807df1e2" path="/var/lib/kubelet/pods/0d0eaf44-dbd7-4248-bfda-e213807df1e2/volumes"
Sep 07 22:47:57.572432 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.572398    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"]
Sep 07 22:47:57.572885 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.572868    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3dbadc0-96b7-416f-9abe-db4c5227379d" containerName="payload-pre-processing"
Sep 07 22:47:57.572885 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.572886    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3dbadc0-96b7-416f-9abe-db4c5227379d" containerName="payload-pre-processing"
Sep 07 22:47:57.572993 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.572901    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d0eaf44-dbd7-4248-bfda-e213807df1e2" containerName="payload-processing"
Sep 07 22:47:57.572993 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.572914    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d0eaf44-dbd7-4248-bfda-e213807df1e2" containerName="payload-processing"
Sep 07 22:47:57.572993 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.572925    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a" containerName="payload-processing"
Sep 07 22:47:57.572993 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.572931    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a" containerName="payload-processing"
Sep 07 22:47:57.572993 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.572941    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d324675-10ff-4300-9187-af1e8d9a2e94" containerName="maas-api"
Sep 07 22:47:57.572993 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.572946    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d324675-10ff-4300-9187-af1e8d9a2e94" containerName="maas-api"
Sep 07 22:47:57.573213 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.573028    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2693deeb-20ad-4b52-bd3a-c64fc0b9ff1a" containerName="payload-processing"
Sep 07 22:47:57.573213 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.573039    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d0eaf44-dbd7-4248-bfda-e213807df1e2" containerName="payload-processing"
Sep 07 22:47:57.573213 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.573047    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d324675-10ff-4300-9187-af1e8d9a2e94" containerName="maas-api"
Sep 07 22:47:57.573213 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.573056    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3dbadc0-96b7-416f-9abe-db4c5227379d" containerName="payload-pre-processing"
Sep 07 22:47:57.577578 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.577554    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:57.579690 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.579669    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-51cb36\"/\"kube-root-ca.crt\""
Sep 07 22:47:57.580426 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.580404    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-51cb36\"/\"default-dockercfg-d5vt6\""
Sep 07 22:47:57.580507 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.580425    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-51cb36\"/\"test-model-51cb36-kserve-self-signed-certs\""
Sep 07 22:47:57.580507 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.580440    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-51cb36\"/\"openshift-service-ca.crt\""
Sep 07 22:47:57.586200 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.586170    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"]
Sep 07 22:47:57.617420 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.617349    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-home\") pod \"test-model-51cb36-kserve-54b8d475f7-g94h7\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") " pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:57.617620 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.617423    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-tmp-dir\") pod \"test-model-51cb36-kserve-54b8d475f7-g94h7\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") " pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:57.617620 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.617490    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-model-cache\") pod \"test-model-51cb36-kserve-54b8d475f7-g94h7\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") " pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:57.617620 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.617518    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f6c0375-54c3-464a-b2a5-6abe0054678c-tls-certs\") pod \"test-model-51cb36-kserve-54b8d475f7-g94h7\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") " pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:57.617620 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.617590    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-dshm\") pod \"test-model-51cb36-kserve-54b8d475f7-g94h7\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") " pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:57.617620 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.617613    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hlp9\" (UniqueName: \"kubernetes.io/projected/4f6c0375-54c3-464a-b2a5-6abe0054678c-kube-api-access-6hlp9\") pod \"test-model-51cb36-kserve-54b8d475f7-g94h7\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") " pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:57.718548 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.718507    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-home\") pod \"test-model-51cb36-kserve-54b8d475f7-g94h7\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") " pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:57.718754 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.718569    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-tmp-dir\") pod \"test-model-51cb36-kserve-54b8d475f7-g94h7\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") " pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:57.718754 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.718615    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-model-cache\") pod \"test-model-51cb36-kserve-54b8d475f7-g94h7\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") " pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:57.718754 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.718636    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f6c0375-54c3-464a-b2a5-6abe0054678c-tls-certs\") pod \"test-model-51cb36-kserve-54b8d475f7-g94h7\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") " pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:57.718754 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.718696    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-dshm\") pod \"test-model-51cb36-kserve-54b8d475f7-g94h7\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") " pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:57.718754 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.718712    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hlp9\" (UniqueName: \"kubernetes.io/projected/4f6c0375-54c3-464a-b2a5-6abe0054678c-kube-api-access-6hlp9\") pod \"test-model-51cb36-kserve-54b8d475f7-g94h7\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") " pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:57.719048 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.718989    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-home\") pod \"test-model-51cb36-kserve-54b8d475f7-g94h7\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") " pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:57.719048 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.719025    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-model-cache\") pod \"test-model-51cb36-kserve-54b8d475f7-g94h7\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") " pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:57.719126 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.719024    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-tmp-dir\") pod \"test-model-51cb36-kserve-54b8d475f7-g94h7\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") " pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:57.721124 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.721090    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-dshm\") pod \"test-model-51cb36-kserve-54b8d475f7-g94h7\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") " pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:57.721239 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.721219    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f6c0375-54c3-464a-b2a5-6abe0054678c-tls-certs\") pod \"test-model-51cb36-kserve-54b8d475f7-g94h7\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") " pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:57.726441 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.726416    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hlp9\" (UniqueName: \"kubernetes.io/projected/4f6c0375-54c3-464a-b2a5-6abe0054678c-kube-api-access-6hlp9\") pod \"test-model-51cb36-kserve-54b8d475f7-g94h7\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") " pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:57.888855 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:57.888812    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:47:58.164774 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:58.164324    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"]
Sep 07 22:47:58.167362 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:47:58.167332    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f6c0375_54c3_464a_b2a5_6abe0054678c.slice/crio-735df3c1fdb432dd79954e9cb152a98c966c3ea99f694179add473d8874079c3 WatchSource:0}: Error finding container 735df3c1fdb432dd79954e9cb152a98c966c3ea99f694179add473d8874079c3: Status 404 returned error can't find the container with id 735df3c1fdb432dd79954e9cb152a98c966c3ea99f694179add473d8874079c3
Sep 07 22:47:58.264557 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:58.264513    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7" event={"ID":"4f6c0375-54c3-464a-b2a5-6abe0054678c","Type":"ContainerStarted","Data":"c7b2bcd0e8b3aeb6996032cc6bcefd6f9c72f1f937c8ac734ca2e123a7dba864"}
Sep 07 22:47:58.264557 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:58.264561    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7" event={"ID":"4f6c0375-54c3-464a-b2a5-6abe0054678c","Type":"ContainerStarted","Data":"735df3c1fdb432dd79954e9cb152a98c966c3ea99f694179add473d8874079c3"}
Sep 07 22:47:58.282914 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:47:58.282839    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7" podStartSLOduration=1.282823134 podStartE2EDuration="1.282823134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:47:58.280662167 +0000 UTC m=+961.337495129" watchObservedRunningTime="2026-09-07 22:47:58.282823134 +0000 UTC m=+961.339656096"
Sep 07 22:48:07.889745 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:07.889707    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:48:07.889745 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:07.889755    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:48:07.902687 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:07.902657    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:48:08.326951 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:08.326913    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:48:14.445818 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:14.445777    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk"]
Sep 07 22:48:14.449424 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:14.449404    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk"
Sep 07 22:48:14.451550 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:14.451522    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-52b103\""
Sep 07 22:48:14.451764 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:14.451748    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-52b103-dockercfg-bnfqz\""
Sep 07 22:48:14.458658 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:14.458631    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk"]
Sep 07 22:48:14.581835 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:14.581790    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d61156b-c8fb-4de7-bcab-667c6927458d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk\" (UID: \"3d61156b-c8fb-4de7-bcab-667c6927458d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk"
Sep 07 22:48:14.582024 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:14.581845    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8vl5\" (UniqueName: \"kubernetes.io/projected/3d61156b-c8fb-4de7-bcab-667c6927458d-kube-api-access-g8vl5\") pod \"payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk\" (UID: \"3d61156b-c8fb-4de7-bcab-667c6927458d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk"
Sep 07 22:48:14.682783 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:14.682741    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d61156b-c8fb-4de7-bcab-667c6927458d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk\" (UID: \"3d61156b-c8fb-4de7-bcab-667c6927458d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk"
Sep 07 22:48:14.682953 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:14.682805    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8vl5\" (UniqueName: \"kubernetes.io/projected/3d61156b-c8fb-4de7-bcab-667c6927458d-kube-api-access-g8vl5\") pod \"payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk\" (UID: \"3d61156b-c8fb-4de7-bcab-667c6927458d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk"
Sep 07 22:48:14.683473 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:14.683451    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d61156b-c8fb-4de7-bcab-667c6927458d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk\" (UID: \"3d61156b-c8fb-4de7-bcab-667c6927458d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk"
Sep 07 22:48:14.690360 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:14.690325    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8vl5\" (UniqueName: \"kubernetes.io/projected/3d61156b-c8fb-4de7-bcab-667c6927458d-kube-api-access-g8vl5\") pod \"payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk\" (UID: \"3d61156b-c8fb-4de7-bcab-667c6927458d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk"
Sep 07 22:48:14.762504 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:14.762409    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk"
Sep 07 22:48:15.050903 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:15.050813    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk"]
Sep 07 22:48:15.054693 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:48:15.054657    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d61156b_c8fb_4de7_bcab_667c6927458d.slice/crio-78ff0310f7ffe01ef5cf4a7a3bab010cbffed6a0051fb5183a18df9eb0c3187a WatchSource:0}: Error finding container 78ff0310f7ffe01ef5cf4a7a3bab010cbffed6a0051fb5183a18df9eb0c3187a: Status 404 returned error can't find the container with id 78ff0310f7ffe01ef5cf4a7a3bab010cbffed6a0051fb5183a18df9eb0c3187a
Sep 07 22:48:15.353935 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:15.353896    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk" event={"ID":"3d61156b-c8fb-4de7-bcab-667c6927458d","Type":"ContainerStarted","Data":"0e4f2c75a3d4a735808a42b6099f352de17c0a3d92b956bb48fe34dc13505895"}
Sep 07 22:48:15.354132 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:15.353957    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk" event={"ID":"3d61156b-c8fb-4de7-bcab-667c6927458d","Type":"ContainerStarted","Data":"78ff0310f7ffe01ef5cf4a7a3bab010cbffed6a0051fb5183a18df9eb0c3187a"}
Sep 07 22:48:15.354132 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:15.353998    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk"
Sep 07 22:48:15.368148 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:15.368091    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk" podStartSLOduration=1.368071038 podStartE2EDuration="1.368071038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:48:15.367208603 +0000 UTC m=+978.424041566" watchObservedRunningTime="2026-09-07 22:48:15.368071038 +0000 UTC m=+978.424904005"
Sep 07 22:48:26.361732 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:48:26.361639    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk"
Sep 07 22:49:28.192852 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.192811    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"]
Sep 07 22:49:28.197151 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.197123    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"
Sep 07 22:49:28.199137 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.199112    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2872fd39-serving-cert\""
Sep 07 22:49:28.206878 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.206850    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"]
Sep 07 22:49:28.286990 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.286952    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/984d1295-2640-4931-8f5e-85632a35a95d-metrics-certs\") pod \"maas-api-e2e-mt-2872fd39-7c587d5948-kp52g\" (UID: \"984d1295-2640-4931-8f5e-85632a35a95d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"
Sep 07 22:49:28.287173 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.287006    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-455nx\" (UniqueName: \"kubernetes.io/projected/984d1295-2640-4931-8f5e-85632a35a95d-kube-api-access-455nx\") pod \"maas-api-e2e-mt-2872fd39-7c587d5948-kp52g\" (UID: \"984d1295-2640-4931-8f5e-85632a35a95d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"
Sep 07 22:49:28.287173 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.287143    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/984d1295-2640-4931-8f5e-85632a35a95d-maas-api-tls\") pod \"maas-api-e2e-mt-2872fd39-7c587d5948-kp52g\" (UID: \"984d1295-2640-4931-8f5e-85632a35a95d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"
Sep 07 22:49:28.287277 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.287179    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/984d1295-2640-4931-8f5e-85632a35a95d-openshift-service-ca\") pod \"maas-api-e2e-mt-2872fd39-7c587d5948-kp52g\" (UID: \"984d1295-2640-4931-8f5e-85632a35a95d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"
Sep 07 22:49:28.388303 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.388231    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/984d1295-2640-4931-8f5e-85632a35a95d-metrics-certs\") pod \"maas-api-e2e-mt-2872fd39-7c587d5948-kp52g\" (UID: \"984d1295-2640-4931-8f5e-85632a35a95d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"
Sep 07 22:49:28.388457 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.388338    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-455nx\" (UniqueName: \"kubernetes.io/projected/984d1295-2640-4931-8f5e-85632a35a95d-kube-api-access-455nx\") pod \"maas-api-e2e-mt-2872fd39-7c587d5948-kp52g\" (UID: \"984d1295-2640-4931-8f5e-85632a35a95d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"
Sep 07 22:49:28.388522 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.388453    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/984d1295-2640-4931-8f5e-85632a35a95d-maas-api-tls\") pod \"maas-api-e2e-mt-2872fd39-7c587d5948-kp52g\" (UID: \"984d1295-2640-4931-8f5e-85632a35a95d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"
Sep 07 22:49:28.388522 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.388484    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/984d1295-2640-4931-8f5e-85632a35a95d-openshift-service-ca\") pod \"maas-api-e2e-mt-2872fd39-7c587d5948-kp52g\" (UID: \"984d1295-2640-4931-8f5e-85632a35a95d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"
Sep 07 22:49:28.389147 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.389120    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/984d1295-2640-4931-8f5e-85632a35a95d-openshift-service-ca\") pod \"maas-api-e2e-mt-2872fd39-7c587d5948-kp52g\" (UID: \"984d1295-2640-4931-8f5e-85632a35a95d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"
Sep 07 22:49:28.390849 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.390822    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/984d1295-2640-4931-8f5e-85632a35a95d-maas-api-tls\") pod \"maas-api-e2e-mt-2872fd39-7c587d5948-kp52g\" (UID: \"984d1295-2640-4931-8f5e-85632a35a95d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"
Sep 07 22:49:28.391003 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.390983    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/984d1295-2640-4931-8f5e-85632a35a95d-metrics-certs\") pod \"maas-api-e2e-mt-2872fd39-7c587d5948-kp52g\" (UID: \"984d1295-2640-4931-8f5e-85632a35a95d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"
Sep 07 22:49:28.397889 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.397864    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-455nx\" (UniqueName: \"kubernetes.io/projected/984d1295-2640-4931-8f5e-85632a35a95d-kube-api-access-455nx\") pod \"maas-api-e2e-mt-2872fd39-7c587d5948-kp52g\" (UID: \"984d1295-2640-4931-8f5e-85632a35a95d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"
Sep 07 22:49:28.512496 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.512399    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"
Sep 07 22:49:28.791346 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.791315    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"]
Sep 07 22:49:28.793786 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:49:28.793762    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984d1295_2640_4931_8f5e_85632a35a95d.slice/crio-e800a4d3176b92d8b11e96b939e2912be3d6f2ffec6a3388c7f4fe15dd3e6817 WatchSource:0}: Error finding container e800a4d3176b92d8b11e96b939e2912be3d6f2ffec6a3388c7f4fe15dd3e6817: Status 404 returned error can't find the container with id e800a4d3176b92d8b11e96b939e2912be3d6f2ffec6a3388c7f4fe15dd3e6817
Sep 07 22:49:28.795155 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.795138    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 22:49:28.944744 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.944707    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5"]
Sep 07 22:49:28.949518 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.949493    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5"
Sep 07 22:49:28.951706 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.951686    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2872fd39\""
Sep 07 22:49:28.951823 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.951805    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2872fd39-dockercfg-85649\""
Sep 07 22:49:28.960354 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:28.960323    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5"]
Sep 07 22:49:29.096380 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:29.096344    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72z7q\" (UniqueName: \"kubernetes.io/projected/f53b4e31-579b-4a9f-b78d-184368a28fc6-kube-api-access-72z7q\") pod \"payload-processing-e2e-mt-2872fd39-664fb47697-tcll5\" (UID: \"f53b4e31-579b-4a9f-b78d-184368a28fc6\") " pod="openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5"
Sep 07 22:49:29.096568 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:29.096534    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53b4e31-579b-4a9f-b78d-184368a28fc6-plugins-config-volume\") pod \"payload-processing-e2e-mt-2872fd39-664fb47697-tcll5\" (UID: \"f53b4e31-579b-4a9f-b78d-184368a28fc6\") " pod="openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5"
Sep 07 22:49:29.197118 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:29.197085    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53b4e31-579b-4a9f-b78d-184368a28fc6-plugins-config-volume\") pod \"payload-processing-e2e-mt-2872fd39-664fb47697-tcll5\" (UID: \"f53b4e31-579b-4a9f-b78d-184368a28fc6\") " pod="openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5"
Sep 07 22:49:29.197549 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:29.197132    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72z7q\" (UniqueName: \"kubernetes.io/projected/f53b4e31-579b-4a9f-b78d-184368a28fc6-kube-api-access-72z7q\") pod \"payload-processing-e2e-mt-2872fd39-664fb47697-tcll5\" (UID: \"f53b4e31-579b-4a9f-b78d-184368a28fc6\") " pod="openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5"
Sep 07 22:49:29.197885 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:29.197864    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53b4e31-579b-4a9f-b78d-184368a28fc6-plugins-config-volume\") pod \"payload-processing-e2e-mt-2872fd39-664fb47697-tcll5\" (UID: \"f53b4e31-579b-4a9f-b78d-184368a28fc6\") " pod="openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5"
Sep 07 22:49:29.204919 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:29.204889    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72z7q\" (UniqueName: \"kubernetes.io/projected/f53b4e31-579b-4a9f-b78d-184368a28fc6-kube-api-access-72z7q\") pod \"payload-processing-e2e-mt-2872fd39-664fb47697-tcll5\" (UID: \"f53b4e31-579b-4a9f-b78d-184368a28fc6\") " pod="openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5"
Sep 07 22:49:29.262006 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:29.261967    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5"
Sep 07 22:49:29.534178 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:29.534095    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5"]
Sep 07 22:49:29.536657 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:49:29.536628    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53b4e31_579b_4a9f_b78d_184368a28fc6.slice/crio-35ff1397f91db251b9eb37c8aaf50ac316be926de8a39f6a7710956ef7ee0226 WatchSource:0}: Error finding container 35ff1397f91db251b9eb37c8aaf50ac316be926de8a39f6a7710956ef7ee0226: Status 404 returned error can't find the container with id 35ff1397f91db251b9eb37c8aaf50ac316be926de8a39f6a7710956ef7ee0226
Sep 07 22:49:29.717391 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:29.717348    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk"]
Sep 07 22:49:29.717648 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:29.717593    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk" podUID="3d61156b-c8fb-4de7-bcab-667c6927458d" containerName="payload-pre-processing" containerID="cri-o://0e4f2c75a3d4a735808a42b6099f352de17c0a3d92b956bb48fe34dc13505895" gracePeriod=30
Sep 07 22:49:29.754099 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:29.754018    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g" event={"ID":"984d1295-2640-4931-8f5e-85632a35a95d","Type":"ContainerStarted","Data":"8cb33948f238e7e28c20a8be32240f1ea7eb8eb94cc33ba49579941685faa0bd"}
Sep 07 22:49:29.754099 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:29.754083    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g" event={"ID":"984d1295-2640-4931-8f5e-85632a35a95d","Type":"ContainerStarted","Data":"e800a4d3176b92d8b11e96b939e2912be3d6f2ffec6a3388c7f4fe15dd3e6817"}
Sep 07 22:49:29.754360 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:29.754240    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"
Sep 07 22:49:29.755978 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:29.755949    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5" event={"ID":"f53b4e31-579b-4a9f-b78d-184368a28fc6","Type":"ContainerStarted","Data":"03ba2dd448cf27ce13f61fe6bf12828bb849164ac6de547cf3f67936082da1f5"}
Sep 07 22:49:29.756132 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:29.755996    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5" event={"ID":"f53b4e31-579b-4a9f-b78d-184368a28fc6","Type":"ContainerStarted","Data":"35ff1397f91db251b9eb37c8aaf50ac316be926de8a39f6a7710956ef7ee0226"}
Sep 07 22:49:29.756132 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:29.756025    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5"
Sep 07 22:49:29.772068 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:29.772005    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g" podStartSLOduration=1.387164971 podStartE2EDuration="1.771982758s" totalImagesPullingTime="384.817787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:49:29.769613118 +0000 UTC m=+1052.826446079" watchObservedRunningTime="2026-09-07 22:49:29.771982758 +0000 UTC m=+1052.828815721"
Sep 07 22:49:29.787487 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:29.787386    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5" podStartSLOduration=1.787371218 podStartE2EDuration="1.787371218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:49:29.785573657 +0000 UTC m=+1052.842406653" watchObservedRunningTime="2026-09-07 22:49:29.787371218 +0000 UTC m=+1052.844204178"
Sep 07 22:49:29.973591 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:29.973563    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk"
Sep 07 22:49:30.107416 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:30.107380    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8vl5\" (UniqueName: \"kubernetes.io/projected/3d61156b-c8fb-4de7-bcab-667c6927458d-kube-api-access-g8vl5\") pod \"3d61156b-c8fb-4de7-bcab-667c6927458d\" (UID: \"3d61156b-c8fb-4de7-bcab-667c6927458d\") "
Sep 07 22:49:30.107627 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:30.107585    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d61156b-c8fb-4de7-bcab-667c6927458d-plugins-config-volume\") pod \"3d61156b-c8fb-4de7-bcab-667c6927458d\" (UID: \"3d61156b-c8fb-4de7-bcab-667c6927458d\") "
Sep 07 22:49:30.107957 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:30.107932    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d61156b-c8fb-4de7-bcab-667c6927458d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d61156b-c8fb-4de7-bcab-667c6927458d" (UID: "3d61156b-c8fb-4de7-bcab-667c6927458d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:49:30.109825 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:30.109798    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d61156b-c8fb-4de7-bcab-667c6927458d-kube-api-access-g8vl5" (OuterVolumeSpecName: "kube-api-access-g8vl5") pod "3d61156b-c8fb-4de7-bcab-667c6927458d" (UID: "3d61156b-c8fb-4de7-bcab-667c6927458d"). InnerVolumeSpecName "kube-api-access-g8vl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:49:30.208722 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:30.208688    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8vl5\" (UniqueName: \"kubernetes.io/projected/3d61156b-c8fb-4de7-bcab-667c6927458d-kube-api-access-g8vl5\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:49:30.208722 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:30.208718    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d61156b-c8fb-4de7-bcab-667c6927458d-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:49:30.762484 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:30.762396    2567 generic.go:356] "Generic (PLEG): container finished" podID="3d61156b-c8fb-4de7-bcab-667c6927458d" containerID="0e4f2c75a3d4a735808a42b6099f352de17c0a3d92b956bb48fe34dc13505895" exitCode=0
Sep 07 22:49:30.762484 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:30.762447    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk"
Sep 07 22:49:30.762677 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:30.762479    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk" event={"ID":"3d61156b-c8fb-4de7-bcab-667c6927458d","Type":"ContainerDied","Data":"0e4f2c75a3d4a735808a42b6099f352de17c0a3d92b956bb48fe34dc13505895"}
Sep 07 22:49:30.762677 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:30.762549    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk" event={"ID":"3d61156b-c8fb-4de7-bcab-667c6927458d","Type":"ContainerDied","Data":"78ff0310f7ffe01ef5cf4a7a3bab010cbffed6a0051fb5183a18df9eb0c3187a"}
Sep 07 22:49:30.762677 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:30.762585    2567 scope.go:117] "RemoveContainer" containerID="0e4f2c75a3d4a735808a42b6099f352de17c0a3d92b956bb48fe34dc13505895"
Sep 07 22:49:30.772088 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:30.772065    2567 scope.go:117] "RemoveContainer" containerID="0e4f2c75a3d4a735808a42b6099f352de17c0a3d92b956bb48fe34dc13505895"
Sep 07 22:49:30.772470 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:49:30.772445    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e4f2c75a3d4a735808a42b6099f352de17c0a3d92b956bb48fe34dc13505895\": container with ID starting with 0e4f2c75a3d4a735808a42b6099f352de17c0a3d92b956bb48fe34dc13505895 not found: ID does not exist" containerID="0e4f2c75a3d4a735808a42b6099f352de17c0a3d92b956bb48fe34dc13505895"
Sep 07 22:49:30.772550 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:30.772477    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e4f2c75a3d4a735808a42b6099f352de17c0a3d92b956bb48fe34dc13505895"} err="failed to get container status \"0e4f2c75a3d4a735808a42b6099f352de17c0a3d92b956bb48fe34dc13505895\": rpc error: code = NotFound desc = could not find container \"0e4f2c75a3d4a735808a42b6099f352de17c0a3d92b956bb48fe34dc13505895\": container with ID starting with 0e4f2c75a3d4a735808a42b6099f352de17c0a3d92b956bb48fe34dc13505895 not found: ID does not exist"
Sep 07 22:49:30.783311 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:30.783284    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk"]
Sep 07 22:49:30.786960 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:30.786934    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-52b103-9bd79d7b6-kxdhk"]
Sep 07 22:49:31.465486 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:31.465448    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d61156b-c8fb-4de7-bcab-667c6927458d" path="/var/lib/kubelet/pods/3d61156b-c8fb-4de7-bcab-667c6927458d/volumes"
Sep 07 22:49:35.768702 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:35.768632    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"
Sep 07 22:49:40.765470 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:49:40.765430    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5"
Sep 07 22:50:09.659814 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:09.659774    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"]
Sep 07 22:50:09.660398 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:09.659996    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g" podUID="984d1295-2640-4931-8f5e-85632a35a95d" containerName="maas-api" containerID="cri-o://8cb33948f238e7e28c20a8be32240f1ea7eb8eb94cc33ba49579941685faa0bd" gracePeriod=30
Sep 07 22:50:09.705474 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:09.705431    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5"]
Sep 07 22:50:09.705731 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:09.705699    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5" podUID="f53b4e31-579b-4a9f-b78d-184368a28fc6" containerName="payload-processing" containerID="cri-o://03ba2dd448cf27ce13f61fe6bf12828bb849164ac6de547cf3f67936082da1f5" gracePeriod=30
Sep 07 22:50:09.956261 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:09.956221    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"
Sep 07 22:50:09.964708 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:09.964669    2567 generic.go:356] "Generic (PLEG): container finished" podID="984d1295-2640-4931-8f5e-85632a35a95d" containerID="8cb33948f238e7e28c20a8be32240f1ea7eb8eb94cc33ba49579941685faa0bd" exitCode=0
Sep 07 22:50:09.964879 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:09.964720    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"
Sep 07 22:50:09.964879 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:09.964847    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g" event={"ID":"984d1295-2640-4931-8f5e-85632a35a95d","Type":"ContainerDied","Data":"8cb33948f238e7e28c20a8be32240f1ea7eb8eb94cc33ba49579941685faa0bd"}
Sep 07 22:50:09.965006 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:09.964909    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g" event={"ID":"984d1295-2640-4931-8f5e-85632a35a95d","Type":"ContainerDied","Data":"e800a4d3176b92d8b11e96b939e2912be3d6f2ffec6a3388c7f4fe15dd3e6817"}
Sep 07 22:50:09.965006 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:09.964939    2567 scope.go:117] "RemoveContainer" containerID="8cb33948f238e7e28c20a8be32240f1ea7eb8eb94cc33ba49579941685faa0bd"
Sep 07 22:50:09.966839 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:09.966805    2567 generic.go:356] "Generic (PLEG): container finished" podID="f53b4e31-579b-4a9f-b78d-184368a28fc6" containerID="03ba2dd448cf27ce13f61fe6bf12828bb849164ac6de547cf3f67936082da1f5" exitCode=0
Sep 07 22:50:09.967047 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:09.967024    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5" event={"ID":"f53b4e31-579b-4a9f-b78d-184368a28fc6","Type":"ContainerDied","Data":"03ba2dd448cf27ce13f61fe6bf12828bb849164ac6de547cf3f67936082da1f5"}
Sep 07 22:50:09.985570 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:09.985540    2567 scope.go:117] "RemoveContainer" containerID="8cb33948f238e7e28c20a8be32240f1ea7eb8eb94cc33ba49579941685faa0bd"
Sep 07 22:50:09.985880 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:50:09.985855    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb33948f238e7e28c20a8be32240f1ea7eb8eb94cc33ba49579941685faa0bd\": container with ID starting with 8cb33948f238e7e28c20a8be32240f1ea7eb8eb94cc33ba49579941685faa0bd not found: ID does not exist" containerID="8cb33948f238e7e28c20a8be32240f1ea7eb8eb94cc33ba49579941685faa0bd"
Sep 07 22:50:09.985985 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:09.985893    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb33948f238e7e28c20a8be32240f1ea7eb8eb94cc33ba49579941685faa0bd"} err="failed to get container status \"8cb33948f238e7e28c20a8be32240f1ea7eb8eb94cc33ba49579941685faa0bd\": rpc error: code = NotFound desc = could not find container \"8cb33948f238e7e28c20a8be32240f1ea7eb8eb94cc33ba49579941685faa0bd\": container with ID starting with 8cb33948f238e7e28c20a8be32240f1ea7eb8eb94cc33ba49579941685faa0bd not found: ID does not exist"
Sep 07 22:50:09.989748 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:09.989727    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5"
Sep 07 22:50:10.086590 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.086540    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/984d1295-2640-4931-8f5e-85632a35a95d-openshift-service-ca\") pod \"984d1295-2640-4931-8f5e-85632a35a95d\" (UID: \"984d1295-2640-4931-8f5e-85632a35a95d\") "
Sep 07 22:50:10.086777 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.086610    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72z7q\" (UniqueName: \"kubernetes.io/projected/f53b4e31-579b-4a9f-b78d-184368a28fc6-kube-api-access-72z7q\") pod \"f53b4e31-579b-4a9f-b78d-184368a28fc6\" (UID: \"f53b4e31-579b-4a9f-b78d-184368a28fc6\") "
Sep 07 22:50:10.086777 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.086658    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/984d1295-2640-4931-8f5e-85632a35a95d-maas-api-tls\") pod \"984d1295-2640-4931-8f5e-85632a35a95d\" (UID: \"984d1295-2640-4931-8f5e-85632a35a95d\") "
Sep 07 22:50:10.086777 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.086750    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/984d1295-2640-4931-8f5e-85632a35a95d-metrics-certs\") pod \"984d1295-2640-4931-8f5e-85632a35a95d\" (UID: \"984d1295-2640-4931-8f5e-85632a35a95d\") "
Sep 07 22:50:10.086909 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.086791    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-455nx\" (UniqueName: \"kubernetes.io/projected/984d1295-2640-4931-8f5e-85632a35a95d-kube-api-access-455nx\") pod \"984d1295-2640-4931-8f5e-85632a35a95d\" (UID: \"984d1295-2640-4931-8f5e-85632a35a95d\") "
Sep 07 22:50:10.086909 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.086818    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53b4e31-579b-4a9f-b78d-184368a28fc6-plugins-config-volume\") pod \"f53b4e31-579b-4a9f-b78d-184368a28fc6\" (UID: \"f53b4e31-579b-4a9f-b78d-184368a28fc6\") "
Sep 07 22:50:10.087020 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.086978    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984d1295-2640-4931-8f5e-85632a35a95d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "984d1295-2640-4931-8f5e-85632a35a95d" (UID: "984d1295-2640-4931-8f5e-85632a35a95d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:50:10.087343 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.087320    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/984d1295-2640-4931-8f5e-85632a35a95d-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:10.087426 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.087339    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53b4e31-579b-4a9f-b78d-184368a28fc6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f53b4e31-579b-4a9f-b78d-184368a28fc6" (UID: "f53b4e31-579b-4a9f-b78d-184368a28fc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:50:10.088932 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.088896    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/984d1295-2640-4931-8f5e-85632a35a95d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "984d1295-2640-4931-8f5e-85632a35a95d" (UID: "984d1295-2640-4931-8f5e-85632a35a95d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:50:10.089050 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.088982    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984d1295-2640-4931-8f5e-85632a35a95d-kube-api-access-455nx" (OuterVolumeSpecName: "kube-api-access-455nx") pod "984d1295-2640-4931-8f5e-85632a35a95d" (UID: "984d1295-2640-4931-8f5e-85632a35a95d"). InnerVolumeSpecName "kube-api-access-455nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:10.089291 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.089273    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/984d1295-2640-4931-8f5e-85632a35a95d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "984d1295-2640-4931-8f5e-85632a35a95d" (UID: "984d1295-2640-4931-8f5e-85632a35a95d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:50:10.089354 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.089301    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53b4e31-579b-4a9f-b78d-184368a28fc6-kube-api-access-72z7q" (OuterVolumeSpecName: "kube-api-access-72z7q") pod "f53b4e31-579b-4a9f-b78d-184368a28fc6" (UID: "f53b4e31-579b-4a9f-b78d-184368a28fc6"). InnerVolumeSpecName "kube-api-access-72z7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:10.188031 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.187937    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-455nx\" (UniqueName: \"kubernetes.io/projected/984d1295-2640-4931-8f5e-85632a35a95d-kube-api-access-455nx\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:10.188031 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.187970    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53b4e31-579b-4a9f-b78d-184368a28fc6-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:10.188031 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.187983    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72z7q\" (UniqueName: \"kubernetes.io/projected/f53b4e31-579b-4a9f-b78d-184368a28fc6-kube-api-access-72z7q\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:10.188031 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.187993    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/984d1295-2640-4931-8f5e-85632a35a95d-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:10.188031 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.188001    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/984d1295-2640-4931-8f5e-85632a35a95d-metrics-certs\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:10.286999 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.286967    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"]
Sep 07 22:50:10.292313 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.292279    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2872fd39-7c587d5948-kp52g"]
Sep 07 22:50:10.974875 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.974780    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5" event={"ID":"f53b4e31-579b-4a9f-b78d-184368a28fc6","Type":"ContainerDied","Data":"35ff1397f91db251b9eb37c8aaf50ac316be926de8a39f6a7710956ef7ee0226"}
Sep 07 22:50:10.974875 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.974807    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5"
Sep 07 22:50:10.974875 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.974838    2567 scope.go:117] "RemoveContainer" containerID="03ba2dd448cf27ce13f61fe6bf12828bb849164ac6de547cf3f67936082da1f5"
Sep 07 22:50:10.995901 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.995873    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5"]
Sep 07 22:50:10.997716 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:10.997694    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2872fd39-664fb47697-tcll5"]
Sep 07 22:50:11.376773 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:11.376738    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"]
Sep 07 22:50:11.466121 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:11.466089    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="984d1295-2640-4931-8f5e-85632a35a95d" path="/var/lib/kubelet/pods/984d1295-2640-4931-8f5e-85632a35a95d/volumes"
Sep 07 22:50:11.466693 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:11.466674    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f53b4e31-579b-4a9f-b78d-184368a28fc6" path="/var/lib/kubelet/pods/f53b4e31-579b-4a9f-b78d-184368a28fc6/volumes"
Sep 07 22:50:23.105011 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:23.104971    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"]
Sep 07 22:50:23.105531 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:23.105235    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j" podUID="e7cafc2e-4b30-40aa-8b55-343ba65e5c43" containerName="maas-api" containerID="cri-o://e818b1124f201c3df4697ad50fd01ef6f2f5b53f903f9b4367f616495544ee95" gracePeriod=30
Sep 07 22:50:23.371025 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:23.370963    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"
Sep 07 22:50:23.521154 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:23.521115    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-metrics-certs\") pod \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\" (UID: \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\") "
Sep 07 22:50:23.521404 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:23.521181    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-openshift-service-ca\") pod \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\" (UID: \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\") "
Sep 07 22:50:23.521404 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:23.521200    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-maas-api-tls\") pod \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\" (UID: \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\") "
Sep 07 22:50:23.521404 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:23.521225    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j46p\" (UniqueName: \"kubernetes.io/projected/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-kube-api-access-6j46p\") pod \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\" (UID: \"e7cafc2e-4b30-40aa-8b55-343ba65e5c43\") "
Sep 07 22:50:23.521659 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:23.521631    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e7cafc2e-4b30-40aa-8b55-343ba65e5c43" (UID: "e7cafc2e-4b30-40aa-8b55-343ba65e5c43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:50:23.523486 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:23.523458    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e7cafc2e-4b30-40aa-8b55-343ba65e5c43" (UID: "e7cafc2e-4b30-40aa-8b55-343ba65e5c43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:50:23.523486 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:23.523467    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e7cafc2e-4b30-40aa-8b55-343ba65e5c43" (UID: "e7cafc2e-4b30-40aa-8b55-343ba65e5c43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:50:23.523645 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:23.523561    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-kube-api-access-6j46p" (OuterVolumeSpecName: "kube-api-access-6j46p") pod "e7cafc2e-4b30-40aa-8b55-343ba65e5c43" (UID: "e7cafc2e-4b30-40aa-8b55-343ba65e5c43"). InnerVolumeSpecName "kube-api-access-6j46p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:23.622599 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:23.622498    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-metrics-certs\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:23.622599 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:23.622534    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:23.622599 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:23.622544    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:23.622599 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:23.622553    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j46p\" (UniqueName: \"kubernetes.io/projected/e7cafc2e-4b30-40aa-8b55-343ba65e5c43-kube-api-access-6j46p\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:24.048450 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:24.048350    2567 generic.go:356] "Generic (PLEG): container finished" podID="e7cafc2e-4b30-40aa-8b55-343ba65e5c43" containerID="e818b1124f201c3df4697ad50fd01ef6f2f5b53f903f9b4367f616495544ee95" exitCode=0
Sep 07 22:50:24.048450 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:24.048401    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"
Sep 07 22:50:24.048450 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:24.048431    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j" event={"ID":"e7cafc2e-4b30-40aa-8b55-343ba65e5c43","Type":"ContainerDied","Data":"e818b1124f201c3df4697ad50fd01ef6f2f5b53f903f9b4367f616495544ee95"}
Sep 07 22:50:24.048738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:24.048492    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j" event={"ID":"e7cafc2e-4b30-40aa-8b55-343ba65e5c43","Type":"ContainerDied","Data":"4bb811566256c2b3e41c70136db1eeafc5366795c64263b408dff968424a8d12"}
Sep 07 22:50:24.048738 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:24.048532    2567 scope.go:117] "RemoveContainer" containerID="e818b1124f201c3df4697ad50fd01ef6f2f5b53f903f9b4367f616495544ee95"
Sep 07 22:50:24.058946 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:24.058924    2567 scope.go:117] "RemoveContainer" containerID="e818b1124f201c3df4697ad50fd01ef6f2f5b53f903f9b4367f616495544ee95"
Sep 07 22:50:24.059336 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:50:24.059314    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e818b1124f201c3df4697ad50fd01ef6f2f5b53f903f9b4367f616495544ee95\": container with ID starting with e818b1124f201c3df4697ad50fd01ef6f2f5b53f903f9b4367f616495544ee95 not found: ID does not exist" containerID="e818b1124f201c3df4697ad50fd01ef6f2f5b53f903f9b4367f616495544ee95"
Sep 07 22:50:24.059391 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:24.059346    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e818b1124f201c3df4697ad50fd01ef6f2f5b53f903f9b4367f616495544ee95"} err="failed to get container status \"e818b1124f201c3df4697ad50fd01ef6f2f5b53f903f9b4367f616495544ee95\": rpc error: code = NotFound desc = could not find container \"e818b1124f201c3df4697ad50fd01ef6f2f5b53f903f9b4367f616495544ee95\": container with ID starting with e818b1124f201c3df4697ad50fd01ef6f2f5b53f903f9b4367f616495544ee95 not found: ID does not exist"
Sep 07 22:50:24.069799 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:24.069769    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"]
Sep 07 22:50:24.071609 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:24.071590    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51cb36-5d8d77b5cf-rvl4j"]
Sep 07 22:50:25.465746 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:25.465708    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7cafc2e-4b30-40aa-8b55-343ba65e5c43" path="/var/lib/kubelet/pods/e7cafc2e-4b30-40aa-8b55-343ba65e5c43/volumes"
Sep 07 22:50:26.397727 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.397682    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7" podUID="4f6c0375-54c3-464a-b2a5-6abe0054678c" containerName="main" containerID="cri-o://c7b2bcd0e8b3aeb6996032cc6bcefd6f9c72f1f937c8ac734ca2e123a7dba864" gracePeriod=45
Sep 07 22:50:26.661402 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.661337    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:50:26.759931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.759894    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-home\") pod \"4f6c0375-54c3-464a-b2a5-6abe0054678c\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") "
Sep 07 22:50:26.759931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.759947    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hlp9\" (UniqueName: \"kubernetes.io/projected/4f6c0375-54c3-464a-b2a5-6abe0054678c-kube-api-access-6hlp9\") pod \"4f6c0375-54c3-464a-b2a5-6abe0054678c\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") "
Sep 07 22:50:26.760214 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.759970    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f6c0375-54c3-464a-b2a5-6abe0054678c-tls-certs\") pod \"4f6c0375-54c3-464a-b2a5-6abe0054678c\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") "
Sep 07 22:50:26.760214 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.760023    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-model-cache\") pod \"4f6c0375-54c3-464a-b2a5-6abe0054678c\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") "
Sep 07 22:50:26.760214 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.760039    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-tmp-dir\") pod \"4f6c0375-54c3-464a-b2a5-6abe0054678c\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") "
Sep 07 22:50:26.760214 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.760055    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-dshm\") pod \"4f6c0375-54c3-464a-b2a5-6abe0054678c\" (UID: \"4f6c0375-54c3-464a-b2a5-6abe0054678c\") "
Sep 07 22:50:26.760442 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.760214    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-home" (OuterVolumeSpecName: "home") pod "4f6c0375-54c3-464a-b2a5-6abe0054678c" (UID: "4f6c0375-54c3-464a-b2a5-6abe0054678c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:26.760442 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.760343    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f6c0375-54c3-464a-b2a5-6abe0054678c" (UID: "4f6c0375-54c3-464a-b2a5-6abe0054678c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:26.760442 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.760367    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-home\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:26.760442 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.760365    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f6c0375-54c3-464a-b2a5-6abe0054678c" (UID: "4f6c0375-54c3-464a-b2a5-6abe0054678c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:26.762352 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.762317    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f6c0375-54c3-464a-b2a5-6abe0054678c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f6c0375-54c3-464a-b2a5-6abe0054678c" (UID: "4f6c0375-54c3-464a-b2a5-6abe0054678c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:50:26.762481 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.762400    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-dshm" (OuterVolumeSpecName: "dshm") pod "4f6c0375-54c3-464a-b2a5-6abe0054678c" (UID: "4f6c0375-54c3-464a-b2a5-6abe0054678c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:26.762481 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.762406    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6c0375-54c3-464a-b2a5-6abe0054678c-kube-api-access-6hlp9" (OuterVolumeSpecName: "kube-api-access-6hlp9") pod "4f6c0375-54c3-464a-b2a5-6abe0054678c" (UID: "4f6c0375-54c3-464a-b2a5-6abe0054678c"). InnerVolumeSpecName "kube-api-access-6hlp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:26.861664 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.861622    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-model-cache\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:26.861664 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.861654    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-tmp-dir\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:26.861664 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.861662    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f6c0375-54c3-464a-b2a5-6abe0054678c-dshm\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:26.861900 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.861684    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hlp9\" (UniqueName: \"kubernetes.io/projected/4f6c0375-54c3-464a-b2a5-6abe0054678c-kube-api-access-6hlp9\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:26.861900 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:26.861693    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f6c0375-54c3-464a-b2a5-6abe0054678c-tls-certs\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:27.066619 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:27.066584    2567 generic.go:356] "Generic (PLEG): container finished" podID="4f6c0375-54c3-464a-b2a5-6abe0054678c" containerID="c7b2bcd0e8b3aeb6996032cc6bcefd6f9c72f1f937c8ac734ca2e123a7dba864" exitCode=0
Sep 07 22:50:27.066789 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:27.066633    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"
Sep 07 22:50:27.066789 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:27.066664    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7" event={"ID":"4f6c0375-54c3-464a-b2a5-6abe0054678c","Type":"ContainerDied","Data":"c7b2bcd0e8b3aeb6996032cc6bcefd6f9c72f1f937c8ac734ca2e123a7dba864"}
Sep 07 22:50:27.066789 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:27.066703    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7" event={"ID":"4f6c0375-54c3-464a-b2a5-6abe0054678c","Type":"ContainerDied","Data":"735df3c1fdb432dd79954e9cb152a98c966c3ea99f694179add473d8874079c3"}
Sep 07 22:50:27.066789 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:27.066722    2567 scope.go:117] "RemoveContainer" containerID="c7b2bcd0e8b3aeb6996032cc6bcefd6f9c72f1f937c8ac734ca2e123a7dba864"
Sep 07 22:50:27.075924 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:27.075906    2567 scope.go:117] "RemoveContainer" containerID="c7b2bcd0e8b3aeb6996032cc6bcefd6f9c72f1f937c8ac734ca2e123a7dba864"
Sep 07 22:50:27.076239 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:50:27.076218    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7b2bcd0e8b3aeb6996032cc6bcefd6f9c72f1f937c8ac734ca2e123a7dba864\": container with ID starting with c7b2bcd0e8b3aeb6996032cc6bcefd6f9c72f1f937c8ac734ca2e123a7dba864 not found: ID does not exist" containerID="c7b2bcd0e8b3aeb6996032cc6bcefd6f9c72f1f937c8ac734ca2e123a7dba864"
Sep 07 22:50:27.076333 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:27.076270    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b2bcd0e8b3aeb6996032cc6bcefd6f9c72f1f937c8ac734ca2e123a7dba864"} err="failed to get container status \"c7b2bcd0e8b3aeb6996032cc6bcefd6f9c72f1f937c8ac734ca2e123a7dba864\": rpc error: code = NotFound desc = could not find container \"c7b2bcd0e8b3aeb6996032cc6bcefd6f9c72f1f937c8ac734ca2e123a7dba864\": container with ID starting with c7b2bcd0e8b3aeb6996032cc6bcefd6f9c72f1f937c8ac734ca2e123a7dba864 not found: ID does not exist"
Sep 07 22:50:27.086583 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:27.086550    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"]
Sep 07 22:50:27.091005 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:27.090976    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-51cb36/test-model-51cb36-kserve-54b8d475f7-g94h7"]
Sep 07 22:50:27.465902 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:27.465824    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f6c0375-54c3-464a-b2a5-6abe0054678c" path="/var/lib/kubelet/pods/4f6c0375-54c3-464a-b2a5-6abe0054678c/volumes"
Sep 07 22:50:33.580985 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.580944    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"]
Sep 07 22:50:33.581553 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.581532    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f6c0375-54c3-464a-b2a5-6abe0054678c" containerName="main"
Sep 07 22:50:33.581642 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.581557    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f6c0375-54c3-464a-b2a5-6abe0054678c" containerName="main"
Sep 07 22:50:33.581642 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.581589    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="984d1295-2640-4931-8f5e-85632a35a95d" containerName="maas-api"
Sep 07 22:50:33.581642 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.581598    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="984d1295-2640-4931-8f5e-85632a35a95d" containerName="maas-api"
Sep 07 22:50:33.581642 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.581609    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f53b4e31-579b-4a9f-b78d-184368a28fc6" containerName="payload-processing"
Sep 07 22:50:33.581642 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.581618    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f53b4e31-579b-4a9f-b78d-184368a28fc6" containerName="payload-processing"
Sep 07 22:50:33.581642 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.581641    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d61156b-c8fb-4de7-bcab-667c6927458d" containerName="payload-pre-processing"
Sep 07 22:50:33.582001 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.581650    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d61156b-c8fb-4de7-bcab-667c6927458d" containerName="payload-pre-processing"
Sep 07 22:50:33.582001 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.581663    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7cafc2e-4b30-40aa-8b55-343ba65e5c43" containerName="maas-api"
Sep 07 22:50:33.582001 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.581672    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7cafc2e-4b30-40aa-8b55-343ba65e5c43" containerName="maas-api"
Sep 07 22:50:33.582001 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.581766    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f53b4e31-579b-4a9f-b78d-184368a28fc6" containerName="payload-processing"
Sep 07 22:50:33.582001 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.581780    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f6c0375-54c3-464a-b2a5-6abe0054678c" containerName="main"
Sep 07 22:50:33.582001 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.581806    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="984d1295-2640-4931-8f5e-85632a35a95d" containerName="maas-api"
Sep 07 22:50:33.582001 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.581819    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7cafc2e-4b30-40aa-8b55-343ba65e5c43" containerName="maas-api"
Sep 07 22:50:33.582001 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.581829    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d61156b-c8fb-4de7-bcab-667c6927458d" containerName="payload-pre-processing"
Sep 07 22:50:33.585179 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.585156    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"
Sep 07 22:50:33.587277 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.587213    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-82409b-serving-cert\""
Sep 07 22:50:33.593699 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.593526    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"]
Sep 07 22:50:33.621662 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.621625    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c2f6\" (UniqueName: \"kubernetes.io/projected/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-kube-api-access-9c2f6\") pod \"maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x\" (UID: \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"
Sep 07 22:50:33.621829 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.621690    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x\" (UID: \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"
Sep 07 22:50:33.621829 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.621727    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x\" (UID: \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"
Sep 07 22:50:33.621829 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.621765    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x\" (UID: \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"
Sep 07 22:50:33.723023 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.722982    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x\" (UID: \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"
Sep 07 22:50:33.723233 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.723041    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x\" (UID: \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"
Sep 07 22:50:33.723233 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.723110    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c2f6\" (UniqueName: \"kubernetes.io/projected/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-kube-api-access-9c2f6\") pod \"maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x\" (UID: \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"
Sep 07 22:50:33.723233 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.723156    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x\" (UID: \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"
Sep 07 22:50:33.723233 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:50:33.723164    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-serving-cert: secret "maas-api-e2e-shared-a-w2-82409b-serving-cert" not found
Sep 07 22:50:33.723530 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:50:33.723277    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-maas-api-tls podName:564bcfd9-41b6-4ee4-b553-44bccb29a8cd nodeName:}" failed. No retries permitted until 2026-09-07 22:50:34.223229384 +0000 UTC m=+1117.280062327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-maas-api-tls") pod "maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x" (UID: "564bcfd9-41b6-4ee4-b553-44bccb29a8cd") : secret "maas-api-e2e-shared-a-w2-82409b-serving-cert" not found
Sep 07 22:50:33.723754 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.723734    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x\" (UID: \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"
Sep 07 22:50:33.725644 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.725620    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x\" (UID: \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"
Sep 07 22:50:33.733829 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:33.733806    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c2f6\" (UniqueName: \"kubernetes.io/projected/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-kube-api-access-9c2f6\") pod \"maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x\" (UID: \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"
Sep 07 22:50:34.227762 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:34.227723    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x\" (UID: \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"
Sep 07 22:50:34.230453 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:34.230419    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x\" (UID: \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"
Sep 07 22:50:34.322362 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:34.322324    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48"]
Sep 07 22:50:34.326300 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:34.326273    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48"
Sep 07 22:50:34.329030 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:34.328992    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-82409b\""
Sep 07 22:50:34.329161 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:34.329061    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-82409b-dockercfg-fl92x\""
Sep 07 22:50:34.347699 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:34.347663    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48"]
Sep 07 22:50:34.429941 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:34.429903    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11efd1f5-0a20-4c7a-841d-e87b11f492f7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48\" (UID: \"11efd1f5-0a20-4c7a-841d-e87b11f492f7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48"
Sep 07 22:50:34.430090 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:34.430016    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knzcr\" (UniqueName: \"kubernetes.io/projected/11efd1f5-0a20-4c7a-841d-e87b11f492f7-kube-api-access-knzcr\") pod \"payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48\" (UID: \"11efd1f5-0a20-4c7a-841d-e87b11f492f7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48"
Sep 07 22:50:34.500146 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:34.500048    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"
Sep 07 22:50:34.530883 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:34.530839    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11efd1f5-0a20-4c7a-841d-e87b11f492f7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48\" (UID: \"11efd1f5-0a20-4c7a-841d-e87b11f492f7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48"
Sep 07 22:50:34.531044 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:34.530973    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knzcr\" (UniqueName: \"kubernetes.io/projected/11efd1f5-0a20-4c7a-841d-e87b11f492f7-kube-api-access-knzcr\") pod \"payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48\" (UID: \"11efd1f5-0a20-4c7a-841d-e87b11f492f7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48"
Sep 07 22:50:34.531644 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:34.531623    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11efd1f5-0a20-4c7a-841d-e87b11f492f7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48\" (UID: \"11efd1f5-0a20-4c7a-841d-e87b11f492f7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48"
Sep 07 22:50:34.541191 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:34.540183    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knzcr\" (UniqueName: \"kubernetes.io/projected/11efd1f5-0a20-4c7a-841d-e87b11f492f7-kube-api-access-knzcr\") pod \"payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48\" (UID: \"11efd1f5-0a20-4c7a-841d-e87b11f492f7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48"
Sep 07 22:50:34.637897 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:34.637846    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48"
Sep 07 22:50:34.774292 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:34.774240    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"]
Sep 07 22:50:34.776757 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:50:34.776723    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564bcfd9_41b6_4ee4_b553_44bccb29a8cd.slice/crio-706738deb8b5a0803dac3f1083bbd013ec6700ffb6e74ade55bfdfdb81734fc9 WatchSource:0}: Error finding container 706738deb8b5a0803dac3f1083bbd013ec6700ffb6e74ade55bfdfdb81734fc9: Status 404 returned error can't find the container with id 706738deb8b5a0803dac3f1083bbd013ec6700ffb6e74ade55bfdfdb81734fc9
Sep 07 22:50:34.911953 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:34.911923    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48"]
Sep 07 22:50:34.914584 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:50:34.914553    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11efd1f5_0a20_4c7a_841d_e87b11f492f7.slice/crio-2841a78850b6fa9edf5227e68c46238d3d54a145a9b3418aabcf7a57d4a6e064 WatchSource:0}: Error finding container 2841a78850b6fa9edf5227e68c46238d3d54a145a9b3418aabcf7a57d4a6e064: Status 404 returned error can't find the container with id 2841a78850b6fa9edf5227e68c46238d3d54a145a9b3418aabcf7a57d4a6e064
Sep 07 22:50:35.112462 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:35.112424    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x" event={"ID":"564bcfd9-41b6-4ee4-b553-44bccb29a8cd","Type":"ContainerStarted","Data":"706738deb8b5a0803dac3f1083bbd013ec6700ffb6e74ade55bfdfdb81734fc9"}
Sep 07 22:50:35.114324 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:35.114298    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48" event={"ID":"11efd1f5-0a20-4c7a-841d-e87b11f492f7","Type":"ContainerStarted","Data":"21895f16a1067adf6469b651459d2df413c0ee90ff1066c0cada91bb133bf270"}
Sep 07 22:50:35.114490 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:35.114341    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48" event={"ID":"11efd1f5-0a20-4c7a-841d-e87b11f492f7","Type":"ContainerStarted","Data":"2841a78850b6fa9edf5227e68c46238d3d54a145a9b3418aabcf7a57d4a6e064"}
Sep 07 22:50:35.114490 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:35.114434    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48"
Sep 07 22:50:35.133642 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:35.133583    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48" podStartSLOduration=1.133564777 podStartE2EDuration="1.133564777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:50:35.131665341 +0000 UTC m=+1118.188498303" watchObservedRunningTime="2026-09-07 22:50:35.133564777 +0000 UTC m=+1118.190397737"
Sep 07 22:50:36.121610 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:36.121569    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x" event={"ID":"564bcfd9-41b6-4ee4-b553-44bccb29a8cd","Type":"ContainerStarted","Data":"9da03a516473c037839062a98bef980f1a79354aa0a74c06747fde683268ecf0"}
Sep 07 22:50:36.122088 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:36.121712    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"
Sep 07 22:50:36.143787 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:36.143725    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x" podStartSLOduration=2.724432994 podStartE2EDuration="3.143707036s" totalImagesPullingTime="419.274042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:50:36.141658845 +0000 UTC m=+1119.198491808" watchObservedRunningTime="2026-09-07 22:50:36.143707036 +0000 UTC m=+1119.200539999"
Sep 07 22:50:41.075434 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.075396    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"]
Sep 07 22:50:41.076033 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.075610    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2" podUID="f86339cb-5c4a-4e5a-83bc-133659236acf" containerName="maas-api" containerID="cri-o://4f0a805d854c738052dc8397be4f87af7b9c95df460a3d88d9b61a526c0a9fed" gracePeriod=30
Sep 07 22:50:41.131663 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.131627    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk"]
Sep 07 22:50:41.131897 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.131872    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk" podUID="2fed04e2-5c55-4330-b97a-a914df8c899d" containerName="payload-processing" containerID="cri-o://29979e63f30cfb4d9efb4cd49f076189814a1f9d29e752579af7e7c9613c907a" gracePeriod=30
Sep 07 22:50:41.417261 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.417225    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk"
Sep 07 22:50:41.497549 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.497500    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fed04e2-5c55-4330-b97a-a914df8c899d-plugins-config-volume\") pod \"2fed04e2-5c55-4330-b97a-a914df8c899d\" (UID: \"2fed04e2-5c55-4330-b97a-a914df8c899d\") "
Sep 07 22:50:41.497712 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.497649    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwhdt\" (UniqueName: \"kubernetes.io/projected/2fed04e2-5c55-4330-b97a-a914df8c899d-kube-api-access-qwhdt\") pod \"2fed04e2-5c55-4330-b97a-a914df8c899d\" (UID: \"2fed04e2-5c55-4330-b97a-a914df8c899d\") "
Sep 07 22:50:41.497917 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.497892    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fed04e2-5c55-4330-b97a-a914df8c899d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2fed04e2-5c55-4330-b97a-a914df8c899d" (UID: "2fed04e2-5c55-4330-b97a-a914df8c899d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:50:41.500392 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.500357    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fed04e2-5c55-4330-b97a-a914df8c899d-kube-api-access-qwhdt" (OuterVolumeSpecName: "kube-api-access-qwhdt") pod "2fed04e2-5c55-4330-b97a-a914df8c899d" (UID: "2fed04e2-5c55-4330-b97a-a914df8c899d"). InnerVolumeSpecName "kube-api-access-qwhdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:41.552894 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.552864    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"
Sep 07 22:50:41.599290 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.599173    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86339cb-5c4a-4e5a-83bc-133659236acf-openshift-service-ca\") pod \"f86339cb-5c4a-4e5a-83bc-133659236acf\" (UID: \"f86339cb-5c4a-4e5a-83bc-133659236acf\") "
Sep 07 22:50:41.599469 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.599332    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f86339cb-5c4a-4e5a-83bc-133659236acf-metrics-certs\") pod \"f86339cb-5c4a-4e5a-83bc-133659236acf\" (UID: \"f86339cb-5c4a-4e5a-83bc-133659236acf\") "
Sep 07 22:50:41.599469 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.599375    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cwwb\" (UniqueName: \"kubernetes.io/projected/f86339cb-5c4a-4e5a-83bc-133659236acf-kube-api-access-5cwwb\") pod \"f86339cb-5c4a-4e5a-83bc-133659236acf\" (UID: \"f86339cb-5c4a-4e5a-83bc-133659236acf\") "
Sep 07 22:50:41.599469 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.599458    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f86339cb-5c4a-4e5a-83bc-133659236acf-maas-api-tls\") pod \"f86339cb-5c4a-4e5a-83bc-133659236acf\" (UID: \"f86339cb-5c4a-4e5a-83bc-133659236acf\") "
Sep 07 22:50:41.599644 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.599547    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f86339cb-5c4a-4e5a-83bc-133659236acf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f86339cb-5c4a-4e5a-83bc-133659236acf" (UID: "f86339cb-5c4a-4e5a-83bc-133659236acf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:50:41.599747 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.599723    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fed04e2-5c55-4330-b97a-a914df8c899d-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:41.599832 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.599753    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86339cb-5c4a-4e5a-83bc-133659236acf-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:41.599832 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.599770    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwhdt\" (UniqueName: \"kubernetes.io/projected/2fed04e2-5c55-4330-b97a-a914df8c899d-kube-api-access-qwhdt\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:41.601622 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.601589    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f86339cb-5c4a-4e5a-83bc-133659236acf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f86339cb-5c4a-4e5a-83bc-133659236acf" (UID: "f86339cb-5c4a-4e5a-83bc-133659236acf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:50:41.601769 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.601629    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f86339cb-5c4a-4e5a-83bc-133659236acf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f86339cb-5c4a-4e5a-83bc-133659236acf" (UID: "f86339cb-5c4a-4e5a-83bc-133659236acf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:50:41.601769 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.601672    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f86339cb-5c4a-4e5a-83bc-133659236acf-kube-api-access-5cwwb" (OuterVolumeSpecName: "kube-api-access-5cwwb") pod "f86339cb-5c4a-4e5a-83bc-133659236acf" (UID: "f86339cb-5c4a-4e5a-83bc-133659236acf"). InnerVolumeSpecName "kube-api-access-5cwwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:41.701206 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.701161    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f86339cb-5c4a-4e5a-83bc-133659236acf-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:41.701206 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.701196    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f86339cb-5c4a-4e5a-83bc-133659236acf-metrics-certs\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:41.701206 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:41.701205    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cwwb\" (UniqueName: \"kubernetes.io/projected/f86339cb-5c4a-4e5a-83bc-133659236acf-kube-api-access-5cwwb\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:50:42.132081 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:42.132049    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"
Sep 07 22:50:42.153728 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:42.153696    2567 generic.go:356] "Generic (PLEG): container finished" podID="f86339cb-5c4a-4e5a-83bc-133659236acf" containerID="4f0a805d854c738052dc8397be4f87af7b9c95df460a3d88d9b61a526c0a9fed" exitCode=0
Sep 07 22:50:42.153893 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:42.153742    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"
Sep 07 22:50:42.153893 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:42.153767    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2" event={"ID":"f86339cb-5c4a-4e5a-83bc-133659236acf","Type":"ContainerDied","Data":"4f0a805d854c738052dc8397be4f87af7b9c95df460a3d88d9b61a526c0a9fed"}
Sep 07 22:50:42.153893 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:42.153816    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2" event={"ID":"f86339cb-5c4a-4e5a-83bc-133659236acf","Type":"ContainerDied","Data":"8e4672dfe9e4630d64d1afb77a2adb4468eefca0ea5bcf195fa23f49718247c5"}
Sep 07 22:50:42.153893 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:42.153840    2567 scope.go:117] "RemoveContainer" containerID="4f0a805d854c738052dc8397be4f87af7b9c95df460a3d88d9b61a526c0a9fed"
Sep 07 22:50:42.155286 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:42.155267    2567 generic.go:356] "Generic (PLEG): container finished" podID="2fed04e2-5c55-4330-b97a-a914df8c899d" containerID="29979e63f30cfb4d9efb4cd49f076189814a1f9d29e752579af7e7c9613c907a" exitCode=0
Sep 07 22:50:42.155383 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:42.155301    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk" event={"ID":"2fed04e2-5c55-4330-b97a-a914df8c899d","Type":"ContainerDied","Data":"29979e63f30cfb4d9efb4cd49f076189814a1f9d29e752579af7e7c9613c907a"}
Sep 07 22:50:42.155383 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:42.155334    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk" event={"ID":"2fed04e2-5c55-4330-b97a-a914df8c899d","Type":"ContainerDied","Data":"a9b73b8adca3293d00682d2b41cd3af5ca639bc81a017047c35d416244f8f6aa"}
Sep 07 22:50:42.155383 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:42.155343    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk"
Sep 07 22:50:42.172021 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:42.171994    2567 scope.go:117] "RemoveContainer" containerID="4f0a805d854c738052dc8397be4f87af7b9c95df460a3d88d9b61a526c0a9fed"
Sep 07 22:50:42.172597 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:50:42.172568    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0a805d854c738052dc8397be4f87af7b9c95df460a3d88d9b61a526c0a9fed\": container with ID starting with 4f0a805d854c738052dc8397be4f87af7b9c95df460a3d88d9b61a526c0a9fed not found: ID does not exist" containerID="4f0a805d854c738052dc8397be4f87af7b9c95df460a3d88d9b61a526c0a9fed"
Sep 07 22:50:42.172718 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:42.172610    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0a805d854c738052dc8397be4f87af7b9c95df460a3d88d9b61a526c0a9fed"} err="failed to get container status \"4f0a805d854c738052dc8397be4f87af7b9c95df460a3d88d9b61a526c0a9fed\": rpc error: code = NotFound desc = could not find container \"4f0a805d854c738052dc8397be4f87af7b9c95df460a3d88d9b61a526c0a9fed\": container with ID starting with 4f0a805d854c738052dc8397be4f87af7b9c95df460a3d88d9b61a526c0a9fed not found: ID does not exist"
Sep 07 22:50:42.172718 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:42.172633    2567 scope.go:117] "RemoveContainer" containerID="29979e63f30cfb4d9efb4cd49f076189814a1f9d29e752579af7e7c9613c907a"
Sep 07 22:50:42.184079 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:42.183890    2567 scope.go:117] "RemoveContainer" containerID="29979e63f30cfb4d9efb4cd49f076189814a1f9d29e752579af7e7c9613c907a"
Sep 07 22:50:42.184381 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:50:42.184350    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29979e63f30cfb4d9efb4cd49f076189814a1f9d29e752579af7e7c9613c907a\": container with ID starting with 29979e63f30cfb4d9efb4cd49f076189814a1f9d29e752579af7e7c9613c907a not found: ID does not exist" containerID="29979e63f30cfb4d9efb4cd49f076189814a1f9d29e752579af7e7c9613c907a"
Sep 07 22:50:42.184608 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:42.184389    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29979e63f30cfb4d9efb4cd49f076189814a1f9d29e752579af7e7c9613c907a"} err="failed to get container status \"29979e63f30cfb4d9efb4cd49f076189814a1f9d29e752579af7e7c9613c907a\": rpc error: code = NotFound desc = could not find container \"29979e63f30cfb4d9efb4cd49f076189814a1f9d29e752579af7e7c9613c907a\": container with ID starting with 29979e63f30cfb4d9efb4cd49f076189814a1f9d29e752579af7e7c9613c907a not found: ID does not exist"
Sep 07 22:50:42.190412 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:42.190383    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"]
Sep 07 22:50:42.194188 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:42.194163    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2"]
Sep 07 22:50:42.205295 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:42.205266    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk"]
Sep 07 22:50:42.209205 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:42.209178    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-79b9aa-7f487fdf7c-l5ntk"]
Sep 07 22:50:43.466553 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:43.466514    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fed04e2-5c55-4330-b97a-a914df8c899d" path="/var/lib/kubelet/pods/2fed04e2-5c55-4330-b97a-a914df8c899d/volumes"
Sep 07 22:50:43.467117 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:43.467091    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f86339cb-5c4a-4e5a-83bc-133659236acf" path="/var/lib/kubelet/pods/f86339cb-5c4a-4e5a-83bc-133659236acf/volumes"
Sep 07 22:50:44.498479 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:44.498423    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79b9aa-65659fb7bf-8nrk2" podUID="f86339cb-5c4a-4e5a-83bc-133659236acf" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.48:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 07 22:50:46.124109 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:46.124075    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48"
Sep 07 22:50:53.367169 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:53.367135    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx"]
Sep 07 22:50:53.367587 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:53.367533    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f86339cb-5c4a-4e5a-83bc-133659236acf" containerName="maas-api"
Sep 07 22:50:53.367587 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:53.367545    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f86339cb-5c4a-4e5a-83bc-133659236acf" containerName="maas-api"
Sep 07 22:50:53.367587 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:53.367565    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fed04e2-5c55-4330-b97a-a914df8c899d" containerName="payload-processing"
Sep 07 22:50:53.367587 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:53.367571    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fed04e2-5c55-4330-b97a-a914df8c899d" containerName="payload-processing"
Sep 07 22:50:53.367722 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:53.367630    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f86339cb-5c4a-4e5a-83bc-133659236acf" containerName="maas-api"
Sep 07 22:50:53.367722 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:53.367639    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fed04e2-5c55-4330-b97a-a914df8c899d" containerName="payload-processing"
Sep 07 22:50:53.371621 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:53.371600    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx"
Sep 07 22:50:53.373758 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:53.373734    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-1d0395-dockercfg-tbh7w\""
Sep 07 22:50:53.374183 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:53.374161    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-1d0395\""
Sep 07 22:50:53.379856 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:53.379597    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx"]
Sep 07 22:50:53.516679 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:53.516638    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75928\" (UniqueName: \"kubernetes.io/projected/2c6c3708-549a-4e52-87fa-8ae773a62bdd-kube-api-access-75928\") pod \"payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx\" (UID: \"2c6c3708-549a-4e52-87fa-8ae773a62bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx"
Sep 07 22:50:53.516852 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:53.516779    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6c3708-549a-4e52-87fa-8ae773a62bdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx\" (UID: \"2c6c3708-549a-4e52-87fa-8ae773a62bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx"
Sep 07 22:50:53.618338 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:53.618226    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6c3708-549a-4e52-87fa-8ae773a62bdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx\" (UID: \"2c6c3708-549a-4e52-87fa-8ae773a62bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx"
Sep 07 22:50:53.618338 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:53.618322    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75928\" (UniqueName: \"kubernetes.io/projected/2c6c3708-549a-4e52-87fa-8ae773a62bdd-kube-api-access-75928\") pod \"payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx\" (UID: \"2c6c3708-549a-4e52-87fa-8ae773a62bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx"
Sep 07 22:50:53.619031 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:53.619007    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6c3708-549a-4e52-87fa-8ae773a62bdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx\" (UID: \"2c6c3708-549a-4e52-87fa-8ae773a62bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx"
Sep 07 22:50:53.625985 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:53.625956    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75928\" (UniqueName: \"kubernetes.io/projected/2c6c3708-549a-4e52-87fa-8ae773a62bdd-kube-api-access-75928\") pod \"payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx\" (UID: \"2c6c3708-549a-4e52-87fa-8ae773a62bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx"
Sep 07 22:50:53.683613 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:53.683577    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx"
Sep 07 22:50:53.953045 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:53.952945    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx"]
Sep 07 22:50:53.956828 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:50:53.956798    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6c3708_549a_4e52_87fa_8ae773a62bdd.slice/crio-9ca6b1ba49bcd9cec399048be9b2ea71d093bdac3df1eee0a1108926e59acea9 WatchSource:0}: Error finding container 9ca6b1ba49bcd9cec399048be9b2ea71d093bdac3df1eee0a1108926e59acea9: Status 404 returned error can't find the container with id 9ca6b1ba49bcd9cec399048be9b2ea71d093bdac3df1eee0a1108926e59acea9
Sep 07 22:50:54.222747 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:54.222664    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx" event={"ID":"2c6c3708-549a-4e52-87fa-8ae773a62bdd","Type":"ContainerStarted","Data":"136c5c52cbeb9240b0b08d475d232b52bc9c5e5778ed1aefa1ffa355b8faafc1"}
Sep 07 22:50:54.222747 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:54.222717    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx" event={"ID":"2c6c3708-549a-4e52-87fa-8ae773a62bdd","Type":"ContainerStarted","Data":"9ca6b1ba49bcd9cec399048be9b2ea71d093bdac3df1eee0a1108926e59acea9"}
Sep 07 22:50:54.222938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:54.222805    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx"
Sep 07 22:50:54.237262 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:50:54.237188    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx" podStartSLOduration=1.237171053 podStartE2EDuration="1.237171053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:50:54.236047124 +0000 UTC m=+1137.292880086" watchObservedRunningTime="2026-09-07 22:50:54.237171053 +0000 UTC m=+1137.294004014"
Sep 07 22:51:02.490256 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.490210    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk"]
Sep 07 22:51:02.495428 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.495402    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk"
Sep 07 22:51:02.497807 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.497780    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-5483b1\""
Sep 07 22:51:02.503337 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.503311    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6"]
Sep 07 22:51:02.507210 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.507187    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk"]
Sep 07 22:51:02.507363 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.507320    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6"
Sep 07 22:51:02.509495 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.509469    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-5483b1-dockercfg-4qp2k\""
Sep 07 22:51:02.519316 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.519284    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6"]
Sep 07 22:51:02.607279 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.607232    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s67hd\" (UniqueName: \"kubernetes.io/projected/5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde-kube-api-access-s67hd\") pod \"payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk\" (UID: \"5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk"
Sep 07 22:51:02.607469 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.607302    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk\" (UID: \"5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk"
Sep 07 22:51:02.607469 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.607352    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zrmd\" (UniqueName: \"kubernetes.io/projected/76aba709-84f3-4332-b14e-5c8f75f5ae66-kube-api-access-9zrmd\") pod \"payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6\" (UID: \"76aba709-84f3-4332-b14e-5c8f75f5ae66\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6"
Sep 07 22:51:02.607469 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.607403    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76aba709-84f3-4332-b14e-5c8f75f5ae66-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6\" (UID: \"76aba709-84f3-4332-b14e-5c8f75f5ae66\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6"
Sep 07 22:51:02.708719 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.708678    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zrmd\" (UniqueName: \"kubernetes.io/projected/76aba709-84f3-4332-b14e-5c8f75f5ae66-kube-api-access-9zrmd\") pod \"payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6\" (UID: \"76aba709-84f3-4332-b14e-5c8f75f5ae66\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6"
Sep 07 22:51:02.708719 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.708725    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76aba709-84f3-4332-b14e-5c8f75f5ae66-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6\" (UID: \"76aba709-84f3-4332-b14e-5c8f75f5ae66\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6"
Sep 07 22:51:02.708995 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.708790    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s67hd\" (UniqueName: \"kubernetes.io/projected/5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde-kube-api-access-s67hd\") pod \"payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk\" (UID: \"5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk"
Sep 07 22:51:02.708995 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.708824    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk\" (UID: \"5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk"
Sep 07 22:51:02.709766 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.709739    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk\" (UID: \"5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk"
Sep 07 22:51:02.709834 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.709739    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76aba709-84f3-4332-b14e-5c8f75f5ae66-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6\" (UID: \"76aba709-84f3-4332-b14e-5c8f75f5ae66\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6"
Sep 07 22:51:02.716811 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.716784    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s67hd\" (UniqueName: \"kubernetes.io/projected/5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde-kube-api-access-s67hd\") pod \"payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk\" (UID: \"5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk"
Sep 07 22:51:02.716937 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.716879    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zrmd\" (UniqueName: \"kubernetes.io/projected/76aba709-84f3-4332-b14e-5c8f75f5ae66-kube-api-access-9zrmd\") pod \"payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6\" (UID: \"76aba709-84f3-4332-b14e-5c8f75f5ae66\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6"
Sep 07 22:51:02.806726 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.806628    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk"
Sep 07 22:51:02.820033 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:02.820003    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6"
Sep 07 22:51:03.082038 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:03.082013    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk"]
Sep 07 22:51:03.083938 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:51:03.083907    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5efcdd91_a3f8_4b84_bcc6_afc8b82f8dde.slice/crio-74989033581a8777a73e21abd5e4c27575de17e517422ec538644503c6b32aff WatchSource:0}: Error finding container 74989033581a8777a73e21abd5e4c27575de17e517422ec538644503c6b32aff: Status 404 returned error can't find the container with id 74989033581a8777a73e21abd5e4c27575de17e517422ec538644503c6b32aff
Sep 07 22:51:03.106322 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:03.106295    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6"]
Sep 07 22:51:03.118454 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:51:03.118421    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76aba709_84f3_4332_b14e_5c8f75f5ae66.slice/crio-781a08a319d0169d6de6589c4f77a77ca668ddfb2c181c9d1c9ff1492fca2387 WatchSource:0}: Error finding container 781a08a319d0169d6de6589c4f77a77ca668ddfb2c181c9d1c9ff1492fca2387: Status 404 returned error can't find the container with id 781a08a319d0169d6de6589c4f77a77ca668ddfb2c181c9d1c9ff1492fca2387
Sep 07 22:51:03.272057 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:03.272019    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6" event={"ID":"76aba709-84f3-4332-b14e-5c8f75f5ae66","Type":"ContainerStarted","Data":"793119d8e98f4dfa41e777526c93871af559ed48fe89c4043cd97c5a1b642f39"}
Sep 07 22:51:03.272304 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:03.272197    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6"
Sep 07 22:51:03.272304 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:03.272239    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6" event={"ID":"76aba709-84f3-4332-b14e-5c8f75f5ae66","Type":"ContainerStarted","Data":"781a08a319d0169d6de6589c4f77a77ca668ddfb2c181c9d1c9ff1492fca2387"}
Sep 07 22:51:03.273761 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:03.273734    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk" event={"ID":"5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde","Type":"ContainerStarted","Data":"fb2aabc868b60e2c9f8fd9dde54120fd97a3720389fd03da59fbd1220719ac34"}
Sep 07 22:51:03.273847 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:03.273783    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk" event={"ID":"5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde","Type":"ContainerStarted","Data":"74989033581a8777a73e21abd5e4c27575de17e517422ec538644503c6b32aff"}
Sep 07 22:51:03.273847 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:03.273818    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk"
Sep 07 22:51:03.287709 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:03.287653    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6" podStartSLOduration=1.28763473 podStartE2EDuration="1.28763473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:51:03.28585091 +0000 UTC m=+1146.342683873" watchObservedRunningTime="2026-09-07 22:51:03.28763473 +0000 UTC m=+1146.344467690"
Sep 07 22:51:03.300967 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:03.300910    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk" podStartSLOduration=1.300893512 podStartE2EDuration="1.300893512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:51:03.300156103 +0000 UTC m=+1146.356989077" watchObservedRunningTime="2026-09-07 22:51:03.300893512 +0000 UTC m=+1146.357726472"
Sep 07 22:51:05.230734 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:05.230702    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx"
Sep 07 22:51:14.281157 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:14.281123    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk"
Sep 07 22:51:14.281791 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:14.281210    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6"
Sep 07 22:51:21.574632 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.574544    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl"]
Sep 07 22:51:21.579391 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.579364    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl"
Sep 07 22:51:21.581700 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.581678    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7823cc\""
Sep 07 22:51:21.581845 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.581730    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7823cc-dockercfg-2789b\""
Sep 07 22:51:21.592446 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.592414    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl"]
Sep 07 22:51:21.600391 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.600229    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr"]
Sep 07 22:51:21.604432 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.604408    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr"
Sep 07 22:51:21.619095 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.619061    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr"]
Sep 07 22:51:21.699652 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.699605    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnsfg\" (UniqueName: \"kubernetes.io/projected/6ab5cdb6-fa92-4787-ba43-2dbaaebfb331-kube-api-access-jnsfg\") pod \"payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr\" (UID: \"6ab5cdb6-fa92-4787-ba43-2dbaaebfb331\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr"
Sep 07 22:51:21.699836 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.699733    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjlct\" (UniqueName: \"kubernetes.io/projected/c0f9b192-ba1b-431c-80e6-d7f35c95ecb8-kube-api-access-bjlct\") pod \"payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl\" (UID: \"c0f9b192-ba1b-431c-80e6-d7f35c95ecb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl"
Sep 07 22:51:21.699918 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.699893    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f9b192-ba1b-431c-80e6-d7f35c95ecb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl\" (UID: \"c0f9b192-ba1b-431c-80e6-d7f35c95ecb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl"
Sep 07 22:51:21.700146 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.699936    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5cdb6-fa92-4787-ba43-2dbaaebfb331-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr\" (UID: \"6ab5cdb6-fa92-4787-ba43-2dbaaebfb331\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr"
Sep 07 22:51:21.800521 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.800475    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnsfg\" (UniqueName: \"kubernetes.io/projected/6ab5cdb6-fa92-4787-ba43-2dbaaebfb331-kube-api-access-jnsfg\") pod \"payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr\" (UID: \"6ab5cdb6-fa92-4787-ba43-2dbaaebfb331\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr"
Sep 07 22:51:21.800724 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.800555    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjlct\" (UniqueName: \"kubernetes.io/projected/c0f9b192-ba1b-431c-80e6-d7f35c95ecb8-kube-api-access-bjlct\") pod \"payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl\" (UID: \"c0f9b192-ba1b-431c-80e6-d7f35c95ecb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl"
Sep 07 22:51:21.800724 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.800673    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f9b192-ba1b-431c-80e6-d7f35c95ecb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl\" (UID: \"c0f9b192-ba1b-431c-80e6-d7f35c95ecb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl"
Sep 07 22:51:21.800724 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.800697    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5cdb6-fa92-4787-ba43-2dbaaebfb331-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr\" (UID: \"6ab5cdb6-fa92-4787-ba43-2dbaaebfb331\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr"
Sep 07 22:51:21.801400 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.801379    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f9b192-ba1b-431c-80e6-d7f35c95ecb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl\" (UID: \"c0f9b192-ba1b-431c-80e6-d7f35c95ecb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl"
Sep 07 22:51:21.801458 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.801386    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5cdb6-fa92-4787-ba43-2dbaaebfb331-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr\" (UID: \"6ab5cdb6-fa92-4787-ba43-2dbaaebfb331\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr"
Sep 07 22:51:21.812528 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.812489    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjlct\" (UniqueName: \"kubernetes.io/projected/c0f9b192-ba1b-431c-80e6-d7f35c95ecb8-kube-api-access-bjlct\") pod \"payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl\" (UID: \"c0f9b192-ba1b-431c-80e6-d7f35c95ecb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl"
Sep 07 22:51:21.812528 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.812509    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnsfg\" (UniqueName: \"kubernetes.io/projected/6ab5cdb6-fa92-4787-ba43-2dbaaebfb331-kube-api-access-jnsfg\") pod \"payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr\" (UID: \"6ab5cdb6-fa92-4787-ba43-2dbaaebfb331\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr"
Sep 07 22:51:21.891216 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.891125    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl"
Sep 07 22:51:21.918636 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:21.918603    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr"
Sep 07 22:51:22.167788 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:22.167606    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl"]
Sep 07 22:51:22.170674 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:51:22.170641    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f9b192_ba1b_431c_80e6_d7f35c95ecb8.slice/crio-5a8caeb69d23629f36217104e47122a550eb1c358ee1935e8720dc1054b904ab WatchSource:0}: Error finding container 5a8caeb69d23629f36217104e47122a550eb1c358ee1935e8720dc1054b904ab: Status 404 returned error can't find the container with id 5a8caeb69d23629f36217104e47122a550eb1c358ee1935e8720dc1054b904ab
Sep 07 22:51:22.207554 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:22.207520    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr"]
Sep 07 22:51:22.210903 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:51:22.210858    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab5cdb6_fa92_4787_ba43_2dbaaebfb331.slice/crio-0de68ed1473f6906ce1b48e6a5869e00d3a722fb0e56def3bef0c7ff3808d4dd WatchSource:0}: Error finding container 0de68ed1473f6906ce1b48e6a5869e00d3a722fb0e56def3bef0c7ff3808d4dd: Status 404 returned error can't find the container with id 0de68ed1473f6906ce1b48e6a5869e00d3a722fb0e56def3bef0c7ff3808d4dd
Sep 07 22:51:22.373175 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:22.373138    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl" event={"ID":"c0f9b192-ba1b-431c-80e6-d7f35c95ecb8","Type":"ContainerStarted","Data":"8c70550af69534debe632133d61b9340d4b7a851a92b51e8166839c56c9de5a1"}
Sep 07 22:51:22.373399 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:22.373200    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl" event={"ID":"c0f9b192-ba1b-431c-80e6-d7f35c95ecb8","Type":"ContainerStarted","Data":"5a8caeb69d23629f36217104e47122a550eb1c358ee1935e8720dc1054b904ab"}
Sep 07 22:51:22.373399 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:22.373230    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl"
Sep 07 22:51:22.374984 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:22.374954    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr" event={"ID":"6ab5cdb6-fa92-4787-ba43-2dbaaebfb331","Type":"ContainerStarted","Data":"2ff0e7f27b12fa178eacd801ce10786cf6af137424e431abe32ced2f53d3eff5"}
Sep 07 22:51:22.375128 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:22.375007    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr" event={"ID":"6ab5cdb6-fa92-4787-ba43-2dbaaebfb331","Type":"ContainerStarted","Data":"0de68ed1473f6906ce1b48e6a5869e00d3a722fb0e56def3bef0c7ff3808d4dd"}
Sep 07 22:51:22.375128 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:22.375057    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr"
Sep 07 22:51:22.388412 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:22.388361    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl" podStartSLOduration=1.388344218 podStartE2EDuration="1.388344218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:51:22.386599448 +0000 UTC m=+1165.443432416" watchObservedRunningTime="2026-09-07 22:51:22.388344218 +0000 UTC m=+1165.445177179"
Sep 07 22:51:22.405086 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:22.405023    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr" podStartSLOduration=1.405001097 podStartE2EDuration="1.405001097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:51:22.403815302 +0000 UTC m=+1165.460648264" watchObservedRunningTime="2026-09-07 22:51:22.405001097 +0000 UTC m=+1165.461834059"
Sep 07 22:51:33.384008 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:33.383975    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr"
Sep 07 22:51:33.384505 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:33.384167    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl"
Sep 07 22:51:44.198817 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:44.198781    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"]
Sep 07 22:51:44.205988 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:44.205956    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"
Sep 07 22:51:44.209428 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:44.209398    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-33bf198b-serving-cert\""
Sep 07 22:51:44.216630 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:44.216596    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"]
Sep 07 22:51:44.324158 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:44.324111    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfad090f-3a47-4cb5-b362-f998655b03b2-openshift-service-ca\") pod \"maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5\" (UID: \"cfad090f-3a47-4cb5-b362-f998655b03b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"
Sep 07 22:51:44.324445 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:44.324168    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfad090f-3a47-4cb5-b362-f998655b03b2-maas-api-tls\") pod \"maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5\" (UID: \"cfad090f-3a47-4cb5-b362-f998655b03b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"
Sep 07 22:51:44.324445 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:44.324325    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x65xl\" (UniqueName: \"kubernetes.io/projected/cfad090f-3a47-4cb5-b362-f998655b03b2-kube-api-access-x65xl\") pod \"maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5\" (UID: \"cfad090f-3a47-4cb5-b362-f998655b03b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"
Sep 07 22:51:44.324549 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:44.324466    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfad090f-3a47-4cb5-b362-f998655b03b2-metrics-certs\") pod \"maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5\" (UID: \"cfad090f-3a47-4cb5-b362-f998655b03b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"
Sep 07 22:51:44.424965 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:44.424925    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfad090f-3a47-4cb5-b362-f998655b03b2-metrics-certs\") pod \"maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5\" (UID: \"cfad090f-3a47-4cb5-b362-f998655b03b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"
Sep 07 22:51:44.424965 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:44.424971    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfad090f-3a47-4cb5-b362-f998655b03b2-openshift-service-ca\") pod \"maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5\" (UID: \"cfad090f-3a47-4cb5-b362-f998655b03b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"
Sep 07 22:51:44.425202 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:44.425009    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfad090f-3a47-4cb5-b362-f998655b03b2-maas-api-tls\") pod \"maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5\" (UID: \"cfad090f-3a47-4cb5-b362-f998655b03b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"
Sep 07 22:51:44.425202 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:44.425061    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x65xl\" (UniqueName: \"kubernetes.io/projected/cfad090f-3a47-4cb5-b362-f998655b03b2-kube-api-access-x65xl\") pod \"maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5\" (UID: \"cfad090f-3a47-4cb5-b362-f998655b03b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"
Sep 07 22:51:44.425660 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:44.425638    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfad090f-3a47-4cb5-b362-f998655b03b2-openshift-service-ca\") pod \"maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5\" (UID: \"cfad090f-3a47-4cb5-b362-f998655b03b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"
Sep 07 22:51:44.427499 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:44.427471    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfad090f-3a47-4cb5-b362-f998655b03b2-metrics-certs\") pod \"maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5\" (UID: \"cfad090f-3a47-4cb5-b362-f998655b03b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"
Sep 07 22:51:44.427606 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:44.427524    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfad090f-3a47-4cb5-b362-f998655b03b2-maas-api-tls\") pod \"maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5\" (UID: \"cfad090f-3a47-4cb5-b362-f998655b03b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"
Sep 07 22:51:44.434314 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:44.434285    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x65xl\" (UniqueName: \"kubernetes.io/projected/cfad090f-3a47-4cb5-b362-f998655b03b2-kube-api-access-x65xl\") pod \"maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5\" (UID: \"cfad090f-3a47-4cb5-b362-f998655b03b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"
Sep 07 22:51:44.521573 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:44.521485    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"
Sep 07 22:51:44.800408 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:44.800381    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"]
Sep 07 22:51:44.803098 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:51:44.803056    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfad090f_3a47_4cb5_b362_f998655b03b2.slice/crio-ffbe0e9411c377b15b9c8d0a54ff195580d62145be65cb191cbf3356dda23214 WatchSource:0}: Error finding container ffbe0e9411c377b15b9c8d0a54ff195580d62145be65cb191cbf3356dda23214: Status 404 returned error can't find the container with id ffbe0e9411c377b15b9c8d0a54ff195580d62145be65cb191cbf3356dda23214
Sep 07 22:51:45.497634 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:45.497539    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5" event={"ID":"cfad090f-3a47-4cb5-b362-f998655b03b2","Type":"ContainerStarted","Data":"3f955719cfd32aeb70ab5a6e9224e72385f5349228e00f0b20febd70fe612132"}
Sep 07 22:51:45.497634 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:45.497597    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5" event={"ID":"cfad090f-3a47-4cb5-b362-f998655b03b2","Type":"ContainerStarted","Data":"ffbe0e9411c377b15b9c8d0a54ff195580d62145be65cb191cbf3356dda23214"}
Sep 07 22:51:45.498061 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:45.497682    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"
Sep 07 22:51:45.515683 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:45.515623    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5" podStartSLOduration=1.129065068 podStartE2EDuration="1.515604455s" totalImagesPullingTime="386.539387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:51:45.514560266 +0000 UTC m=+1188.571393268" watchObservedRunningTime="2026-09-07 22:51:45.515604455 +0000 UTC m=+1188.572437416"
Sep 07 22:51:51.508820 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:51.508790    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"
Sep 07 22:51:57.470062 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:57.470030    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/2.log"
Sep 07 22:51:57.471521 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:57.471499    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/2.log"
Sep 07 22:51:57.596202 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:57.596166    2567 scope.go:117] "RemoveContainer" containerID="dfc24cddc2e10e20545c54a5f9b5981401cd4292b70e3ff0098cd38e30621ee2"
Sep 07 22:51:57.605114 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:51:57.605091    2567 scope.go:117] "RemoveContainer" containerID="dce8cf5292582866b5da3744f7e3ab07fda009f032d753061322853594faea96"
Sep 07 22:52:14.067014 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.066978    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"]
Sep 07 22:52:14.072536 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.072504    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.075666 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.075632    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-33bf198b\"/\"openshift-service-ca.crt\""
Sep 07 22:52:14.075824 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.075671    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-33bf198b\"/\"default-dockercfg-n8vdc\""
Sep 07 22:52:14.075824 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.075703    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-33bf198b\"/\"e2e-lifecycle-model-33bf198b-kserve-self-signed-certs\""
Sep 07 22:52:14.076398 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.075998    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-33bf198b\"/\"kube-root-ca.crt\""
Sep 07 22:52:14.077471 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.077447    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"]
Sep 07 22:52:14.227970 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.227916    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd47ca71-b053-42ae-af87-a7ceedd979e4-tls-certs\") pod \"e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") " pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.227970 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.227969    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-home\") pod \"e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") " pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.228294 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.228035    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-dshm\") pod \"e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") " pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.228294 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.228082    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-model-cache\") pod \"e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") " pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.228294 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.228236    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg9fd\" (UniqueName: \"kubernetes.io/projected/cd47ca71-b053-42ae-af87-a7ceedd979e4-kube-api-access-kg9fd\") pod \"e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") " pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.228452 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.228331    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-tmp-dir\") pod \"e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") " pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.329806 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.329699    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd47ca71-b053-42ae-af87-a7ceedd979e4-tls-certs\") pod \"e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") " pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.329806 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.329743    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-home\") pod \"e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") " pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.329806 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.329769    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-dshm\") pod \"e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") " pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.329806 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.329795    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-model-cache\") pod \"e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") " pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.330208 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.329876    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg9fd\" (UniqueName: \"kubernetes.io/projected/cd47ca71-b053-42ae-af87-a7ceedd979e4-kube-api-access-kg9fd\") pod \"e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") " pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.330208 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.329916    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-tmp-dir\") pod \"e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") " pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.330208 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.330191    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-home\") pod \"e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") " pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.330401 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.330217    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-model-cache\") pod \"e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") " pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.330401 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.330315    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-tmp-dir\") pod \"e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") " pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.332125 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.332099    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-dshm\") pod \"e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") " pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.332360 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.332341    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd47ca71-b053-42ae-af87-a7ceedd979e4-tls-certs\") pod \"e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") " pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.337595 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.337570    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg9fd\" (UniqueName: \"kubernetes.io/projected/cd47ca71-b053-42ae-af87-a7ceedd979e4-kube-api-access-kg9fd\") pod \"e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") " pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.383636 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.383596    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:14.659517 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:14.659491    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"]
Sep 07 22:52:14.662216 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:52:14.662183    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd47ca71_b053_42ae_af87_a7ceedd979e4.slice/crio-243f2d50c67e3113d6c0d663c28d77fbbd87128839827c89d3d6509aceaa23f0 WatchSource:0}: Error finding container 243f2d50c67e3113d6c0d663c28d77fbbd87128839827c89d3d6509aceaa23f0: Status 404 returned error can't find the container with id 243f2d50c67e3113d6c0d663c28d77fbbd87128839827c89d3d6509aceaa23f0
Sep 07 22:52:15.655997 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:15.655961    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96" event={"ID":"cd47ca71-b053-42ae-af87-a7ceedd979e4","Type":"ContainerStarted","Data":"011dcd5056da0c6d95e4738b9d5954780e14d35be7714ce62c21ae6b0f1bea60"}
Sep 07 22:52:15.655997 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:15.656001    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96" event={"ID":"cd47ca71-b053-42ae-af87-a7ceedd979e4","Type":"ContainerStarted","Data":"243f2d50c67e3113d6c0d663c28d77fbbd87128839827c89d3d6509aceaa23f0"}
Sep 07 22:52:15.670622 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:15.670563    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96" podStartSLOduration=1.670544909 podStartE2EDuration="1.670544909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:52:15.669989758 +0000 UTC m=+1218.726822714" watchObservedRunningTime="2026-09-07 22:52:15.670544909 +0000 UTC m=+1218.727377870"
Sep 07 22:52:24.384146 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:24.384105    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:24.384593 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:24.384173    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:24.396990 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:24.396953    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:52:24.715656 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:52:24.715567    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:53:33.477456 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:33.477423    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"]
Sep 07 22:53:33.478339 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:33.478289    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5" podUID="cfad090f-3a47-4cb5-b362-f998655b03b2" containerName="maas-api" containerID="cri-o://3f955719cfd32aeb70ab5a6e9224e72385f5349228e00f0b20febd70fe612132" gracePeriod=30
Sep 07 22:53:33.738671 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:33.738606    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"
Sep 07 22:53:33.825089 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:33.825052    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfad090f-3a47-4cb5-b362-f998655b03b2-maas-api-tls\") pod \"cfad090f-3a47-4cb5-b362-f998655b03b2\" (UID: \"cfad090f-3a47-4cb5-b362-f998655b03b2\") "
Sep 07 22:53:33.825293 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:33.825100    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfad090f-3a47-4cb5-b362-f998655b03b2-openshift-service-ca\") pod \"cfad090f-3a47-4cb5-b362-f998655b03b2\" (UID: \"cfad090f-3a47-4cb5-b362-f998655b03b2\") "
Sep 07 22:53:33.825293 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:33.825128    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfad090f-3a47-4cb5-b362-f998655b03b2-metrics-certs\") pod \"cfad090f-3a47-4cb5-b362-f998655b03b2\" (UID: \"cfad090f-3a47-4cb5-b362-f998655b03b2\") "
Sep 07 22:53:33.825293 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:33.825147    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x65xl\" (UniqueName: \"kubernetes.io/projected/cfad090f-3a47-4cb5-b362-f998655b03b2-kube-api-access-x65xl\") pod \"cfad090f-3a47-4cb5-b362-f998655b03b2\" (UID: \"cfad090f-3a47-4cb5-b362-f998655b03b2\") "
Sep 07 22:53:33.825569 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:33.825523    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfad090f-3a47-4cb5-b362-f998655b03b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cfad090f-3a47-4cb5-b362-f998655b03b2" (UID: "cfad090f-3a47-4cb5-b362-f998655b03b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:53:33.827444 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:33.827409    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfad090f-3a47-4cb5-b362-f998655b03b2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cfad090f-3a47-4cb5-b362-f998655b03b2" (UID: "cfad090f-3a47-4cb5-b362-f998655b03b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:53:33.827797 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:33.827770    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfad090f-3a47-4cb5-b362-f998655b03b2-kube-api-access-x65xl" (OuterVolumeSpecName: "kube-api-access-x65xl") pod "cfad090f-3a47-4cb5-b362-f998655b03b2" (UID: "cfad090f-3a47-4cb5-b362-f998655b03b2"). InnerVolumeSpecName "kube-api-access-x65xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:53:33.827797 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:33.827787    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfad090f-3a47-4cb5-b362-f998655b03b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cfad090f-3a47-4cb5-b362-f998655b03b2" (UID: "cfad090f-3a47-4cb5-b362-f998655b03b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:53:33.926367 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:33.926332    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfad090f-3a47-4cb5-b362-f998655b03b2-metrics-certs\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:53:33.926367 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:33.926364    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x65xl\" (UniqueName: \"kubernetes.io/projected/cfad090f-3a47-4cb5-b362-f998655b03b2-kube-api-access-x65xl\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:53:33.926367 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:33.926376    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfad090f-3a47-4cb5-b362-f998655b03b2-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:53:33.926604 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:33.926385    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfad090f-3a47-4cb5-b362-f998655b03b2-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:53:34.079160 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:34.079125    2567 generic.go:356] "Generic (PLEG): container finished" podID="cfad090f-3a47-4cb5-b362-f998655b03b2" containerID="3f955719cfd32aeb70ab5a6e9224e72385f5349228e00f0b20febd70fe612132" exitCode=0
Sep 07 22:53:34.079377 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:34.079169    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"
Sep 07 22:53:34.079377 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:34.079189    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5" event={"ID":"cfad090f-3a47-4cb5-b362-f998655b03b2","Type":"ContainerDied","Data":"3f955719cfd32aeb70ab5a6e9224e72385f5349228e00f0b20febd70fe612132"}
Sep 07 22:53:34.079377 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:34.079236    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5" event={"ID":"cfad090f-3a47-4cb5-b362-f998655b03b2","Type":"ContainerDied","Data":"ffbe0e9411c377b15b9c8d0a54ff195580d62145be65cb191cbf3356dda23214"}
Sep 07 22:53:34.079377 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:34.079291    2567 scope.go:117] "RemoveContainer" containerID="3f955719cfd32aeb70ab5a6e9224e72385f5349228e00f0b20febd70fe612132"
Sep 07 22:53:34.089298 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:34.089277    2567 scope.go:117] "RemoveContainer" containerID="3f955719cfd32aeb70ab5a6e9224e72385f5349228e00f0b20febd70fe612132"
Sep 07 22:53:34.089624 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:53:34.089600    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f955719cfd32aeb70ab5a6e9224e72385f5349228e00f0b20febd70fe612132\": container with ID starting with 3f955719cfd32aeb70ab5a6e9224e72385f5349228e00f0b20febd70fe612132 not found: ID does not exist" containerID="3f955719cfd32aeb70ab5a6e9224e72385f5349228e00f0b20febd70fe612132"
Sep 07 22:53:34.089698 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:34.089637    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f955719cfd32aeb70ab5a6e9224e72385f5349228e00f0b20febd70fe612132"} err="failed to get container status \"3f955719cfd32aeb70ab5a6e9224e72385f5349228e00f0b20febd70fe612132\": rpc error: code = NotFound desc = could not find container \"3f955719cfd32aeb70ab5a6e9224e72385f5349228e00f0b20febd70fe612132\": container with ID starting with 3f955719cfd32aeb70ab5a6e9224e72385f5349228e00f0b20febd70fe612132 not found: ID does not exist"
Sep 07 22:53:34.099740 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:34.099704    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"]
Sep 07 22:53:34.103559 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:34.103535    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-33bf198b-5485f86f69-s2kz5"]
Sep 07 22:53:35.466302 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:35.466270    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfad090f-3a47-4cb5-b362-f998655b03b2" path="/var/lib/kubelet/pods/cfad090f-3a47-4cb5-b362-f998655b03b2/volumes"
Sep 07 22:53:41.749063 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:41.749028    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"]
Sep 07 22:53:57.148944 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.148899    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96" podUID="cd47ca71-b053-42ae-af87-a7ceedd979e4" containerName="main" containerID="cri-o://011dcd5056da0c6d95e4738b9d5954780e14d35be7714ce62c21ae6b0f1bea60" gracePeriod=45
Sep 07 22:53:57.418178 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.418144    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:53:57.554980 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.554945    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-tmp-dir\") pod \"cd47ca71-b053-42ae-af87-a7ceedd979e4\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") "
Sep 07 22:53:57.554980 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.554983    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-home\") pod \"cd47ca71-b053-42ae-af87-a7ceedd979e4\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") "
Sep 07 22:53:57.555279 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.555003    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg9fd\" (UniqueName: \"kubernetes.io/projected/cd47ca71-b053-42ae-af87-a7ceedd979e4-kube-api-access-kg9fd\") pod \"cd47ca71-b053-42ae-af87-a7ceedd979e4\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") "
Sep 07 22:53:57.555279 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.555049    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-model-cache\") pod \"cd47ca71-b053-42ae-af87-a7ceedd979e4\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") "
Sep 07 22:53:57.555279 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.555074    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-dshm\") pod \"cd47ca71-b053-42ae-af87-a7ceedd979e4\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") "
Sep 07 22:53:57.555279 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.555092    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd47ca71-b053-42ae-af87-a7ceedd979e4-tls-certs\") pod \"cd47ca71-b053-42ae-af87-a7ceedd979e4\" (UID: \"cd47ca71-b053-42ae-af87-a7ceedd979e4\") "
Sep 07 22:53:57.555279 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.555237    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-home" (OuterVolumeSpecName: "home") pod "cd47ca71-b053-42ae-af87-a7ceedd979e4" (UID: "cd47ca71-b053-42ae-af87-a7ceedd979e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:53:57.555526 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.555338    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd47ca71-b053-42ae-af87-a7ceedd979e4" (UID: "cd47ca71-b053-42ae-af87-a7ceedd979e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:53:57.555526 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.555397    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd47ca71-b053-42ae-af87-a7ceedd979e4" (UID: "cd47ca71-b053-42ae-af87-a7ceedd979e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:53:57.555603 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.555556    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-tmp-dir\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:53:57.555603 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.555570    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-home\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:53:57.555685 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.555665    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-model-cache\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:53:57.557432 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.557399    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd47ca71-b053-42ae-af87-a7ceedd979e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd47ca71-b053-42ae-af87-a7ceedd979e4" (UID: "cd47ca71-b053-42ae-af87-a7ceedd979e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:53:57.557757 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.557729    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-dshm" (OuterVolumeSpecName: "dshm") pod "cd47ca71-b053-42ae-af87-a7ceedd979e4" (UID: "cd47ca71-b053-42ae-af87-a7ceedd979e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:53:57.557757 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.557746    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd47ca71-b053-42ae-af87-a7ceedd979e4-kube-api-access-kg9fd" (OuterVolumeSpecName: "kube-api-access-kg9fd") pod "cd47ca71-b053-42ae-af87-a7ceedd979e4" (UID: "cd47ca71-b053-42ae-af87-a7ceedd979e4"). InnerVolumeSpecName "kube-api-access-kg9fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:53:57.656784 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.656705    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg9fd\" (UniqueName: \"kubernetes.io/projected/cd47ca71-b053-42ae-af87-a7ceedd979e4-kube-api-access-kg9fd\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:53:57.656784 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.656736    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd47ca71-b053-42ae-af87-a7ceedd979e4-dshm\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:53:57.656784 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:57.656745    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd47ca71-b053-42ae-af87-a7ceedd979e4-tls-certs\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:53:58.212129 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:58.212096    2567 generic.go:356] "Generic (PLEG): container finished" podID="cd47ca71-b053-42ae-af87-a7ceedd979e4" containerID="011dcd5056da0c6d95e4738b9d5954780e14d35be7714ce62c21ae6b0f1bea60" exitCode=0
Sep 07 22:53:58.212664 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:58.212137    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"
Sep 07 22:53:58.212664 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:58.212184    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96" event={"ID":"cd47ca71-b053-42ae-af87-a7ceedd979e4","Type":"ContainerDied","Data":"011dcd5056da0c6d95e4738b9d5954780e14d35be7714ce62c21ae6b0f1bea60"}
Sep 07 22:53:58.212664 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:58.212230    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96" event={"ID":"cd47ca71-b053-42ae-af87-a7ceedd979e4","Type":"ContainerDied","Data":"243f2d50c67e3113d6c0d663c28d77fbbd87128839827c89d3d6509aceaa23f0"}
Sep 07 22:53:58.212664 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:58.212266    2567 scope.go:117] "RemoveContainer" containerID="011dcd5056da0c6d95e4738b9d5954780e14d35be7714ce62c21ae6b0f1bea60"
Sep 07 22:53:58.222404 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:58.222384    2567 scope.go:117] "RemoveContainer" containerID="011dcd5056da0c6d95e4738b9d5954780e14d35be7714ce62c21ae6b0f1bea60"
Sep 07 22:53:58.222719 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:53:58.222699    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"011dcd5056da0c6d95e4738b9d5954780e14d35be7714ce62c21ae6b0f1bea60\": container with ID starting with 011dcd5056da0c6d95e4738b9d5954780e14d35be7714ce62c21ae6b0f1bea60 not found: ID does not exist" containerID="011dcd5056da0c6d95e4738b9d5954780e14d35be7714ce62c21ae6b0f1bea60"
Sep 07 22:53:58.222792 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:58.222727    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011dcd5056da0c6d95e4738b9d5954780e14d35be7714ce62c21ae6b0f1bea60"} err="failed to get container status \"011dcd5056da0c6d95e4738b9d5954780e14d35be7714ce62c21ae6b0f1bea60\": rpc error: code = NotFound desc = could not find container \"011dcd5056da0c6d95e4738b9d5954780e14d35be7714ce62c21ae6b0f1bea60\": container with ID starting with 011dcd5056da0c6d95e4738b9d5954780e14d35be7714ce62c21ae6b0f1bea60 not found: ID does not exist"
Sep 07 22:53:58.233486 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:58.233454    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"]
Sep 07 22:53:58.236697 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:58.236669    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-33bf198b/e2e-lifecycle-model-33bf198b-kserve-6699dcf795-xvn96"]
Sep 07 22:53:59.467495 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:53:59.467458    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd47ca71-b053-42ae-af87-a7ceedd979e4" path="/var/lib/kubelet/pods/cd47ca71-b053-42ae-af87-a7ceedd979e4/volumes"
Sep 07 22:55:09.663094 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:09.663056    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7581-efbcf7"]
Sep 07 22:55:09.663614 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:09.663528    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfad090f-3a47-4cb5-b362-f998655b03b2" containerName="maas-api"
Sep 07 22:55:09.663614 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:09.663546    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfad090f-3a47-4cb5-b362-f998655b03b2" containerName="maas-api"
Sep 07 22:55:09.663614 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:09.663560    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd47ca71-b053-42ae-af87-a7ceedd979e4" containerName="main"
Sep 07 22:55:09.663614 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:09.663566    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd47ca71-b053-42ae-af87-a7ceedd979e4" containerName="main"
Sep 07 22:55:09.663747 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:09.663637    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd47ca71-b053-42ae-af87-a7ceedd979e4" containerName="main"
Sep 07 22:55:09.663747 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:09.663645    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfad090f-3a47-4cb5-b362-f998655b03b2" containerName="maas-api"
Sep 07 22:55:09.667952 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:09.667932    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7581-efbcf7"
Sep 07 22:55:09.670218 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:09.670169    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b689c\""
Sep 07 22:55:09.672565 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:09.672528    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7581-efbcf7"]
Sep 07 22:55:09.784528 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:09.784488    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmhqw\" (UniqueName: \"kubernetes.io/projected/49cdf3c0-baa4-4834-a23d-e48b7f336893-kube-api-access-jmhqw\") pod \"mt-curl-7581-efbcf7\" (UID: \"49cdf3c0-baa4-4834-a23d-e48b7f336893\") " pod="opendatahub/mt-curl-7581-efbcf7"
Sep 07 22:55:09.885851 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:09.885807    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmhqw\" (UniqueName: \"kubernetes.io/projected/49cdf3c0-baa4-4834-a23d-e48b7f336893-kube-api-access-jmhqw\") pod \"mt-curl-7581-efbcf7\" (UID: \"49cdf3c0-baa4-4834-a23d-e48b7f336893\") " pod="opendatahub/mt-curl-7581-efbcf7"
Sep 07 22:55:09.893408 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:09.893374    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmhqw\" (UniqueName: \"kubernetes.io/projected/49cdf3c0-baa4-4834-a23d-e48b7f336893-kube-api-access-jmhqw\") pod \"mt-curl-7581-efbcf7\" (UID: \"49cdf3c0-baa4-4834-a23d-e48b7f336893\") " pod="opendatahub/mt-curl-7581-efbcf7"
Sep 07 22:55:09.979429 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:09.979338    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7581-efbcf7"
Sep 07 22:55:10.252403 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:10.252374    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7581-efbcf7"]
Sep 07 22:55:10.254589 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:55:10.254551    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49cdf3c0_baa4_4834_a23d_e48b7f336893.slice/crio-773506fa9f113042a88d73bdb791ec670ca74ecab50743bf95bdee0f5663168c WatchSource:0}: Error finding container 773506fa9f113042a88d73bdb791ec670ca74ecab50743bf95bdee0f5663168c: Status 404 returned error can't find the container with id 773506fa9f113042a88d73bdb791ec670ca74ecab50743bf95bdee0f5663168c
Sep 07 22:55:10.255945 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:10.255923    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 22:55:10.593731 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:10.593690    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7581-efbcf7" event={"ID":"49cdf3c0-baa4-4834-a23d-e48b7f336893","Type":"ContainerStarted","Data":"773506fa9f113042a88d73bdb791ec670ca74ecab50743bf95bdee0f5663168c"}
Sep 07 22:55:11.599941 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:11.599908    2567 generic.go:356] "Generic (PLEG): container finished" podID="49cdf3c0-baa4-4834-a23d-e48b7f336893" containerID="9772575c522595b26e4a3eb9c533e46c11729b4ba67a5e9726bb53e92a2b0b38" exitCode=0
Sep 07 22:55:11.600363 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:11.599976    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7581-efbcf7" event={"ID":"49cdf3c0-baa4-4834-a23d-e48b7f336893","Type":"ContainerDied","Data":"9772575c522595b26e4a3eb9c533e46c11729b4ba67a5e9726bb53e92a2b0b38"}
Sep 07 22:55:12.739554 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:12.739519    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7581-efbcf7"
Sep 07 22:55:12.771952 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:12.771915    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7581-efbcf7_49cdf3c0-baa4-4834-a23d-e48b7f336893/mt-curl-7581-efbcf7/0.log"
Sep 07 22:55:12.792257 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:12.792209    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7581-efbcf7"]
Sep 07 22:55:12.796515 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:12.796479    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7581-efbcf7"]
Sep 07 22:55:12.815198 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:12.815157    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmhqw\" (UniqueName: \"kubernetes.io/projected/49cdf3c0-baa4-4834-a23d-e48b7f336893-kube-api-access-jmhqw\") pod \"49cdf3c0-baa4-4834-a23d-e48b7f336893\" (UID: \"49cdf3c0-baa4-4834-a23d-e48b7f336893\") "
Sep 07 22:55:12.817321 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:12.817293    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49cdf3c0-baa4-4834-a23d-e48b7f336893-kube-api-access-jmhqw" (OuterVolumeSpecName: "kube-api-access-jmhqw") pod "49cdf3c0-baa4-4834-a23d-e48b7f336893" (UID: "49cdf3c0-baa4-4834-a23d-e48b7f336893"). InnerVolumeSpecName "kube-api-access-jmhqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:55:12.916872 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:12.916783    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmhqw\" (UniqueName: \"kubernetes.io/projected/49cdf3c0-baa4-4834-a23d-e48b7f336893-kube-api-access-jmhqw\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:55:13.467857 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:13.467819    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49cdf3c0-baa4-4834-a23d-e48b7f336893" path="/var/lib/kubelet/pods/49cdf3c0-baa4-4834-a23d-e48b7f336893/volumes"
Sep 07 22:55:13.611429 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:13.611401    2567 scope.go:117] "RemoveContainer" containerID="9772575c522595b26e4a3eb9c533e46c11729b4ba67a5e9726bb53e92a2b0b38"
Sep 07 22:55:13.611629 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:13.611406    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7581-efbcf7"
Sep 07 22:55:53.645987 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:53.645945    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"]
Sep 07 22:55:53.646488 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:53.646383    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49cdf3c0-baa4-4834-a23d-e48b7f336893" containerName="mt-curl-7581-efbcf7"
Sep 07 22:55:53.646488 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:53.646396    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="49cdf3c0-baa4-4834-a23d-e48b7f336893" containerName="mt-curl-7581-efbcf7"
Sep 07 22:55:53.646488 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:53.646464    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="49cdf3c0-baa4-4834-a23d-e48b7f336893" containerName="mt-curl-7581-efbcf7"
Sep 07 22:55:53.652224 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:53.652196    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"
Sep 07 22:55:53.654603 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:53.654579    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-dd5b43-serving-cert\""
Sep 07 22:55:53.660400 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:53.660370    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"]
Sep 07 22:55:53.820696 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:53.820663    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c444bca-6bc7-4d67-8e29-322ba9b044bd-openshift-service-ca\") pod \"maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz\" (UID: \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"
Sep 07 22:55:53.820696 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:53.820700    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c444bca-6bc7-4d67-8e29-322ba9b044bd-maas-api-tls\") pod \"maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz\" (UID: \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"
Sep 07 22:55:53.820930 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:53.820739    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfnmw\" (UniqueName: \"kubernetes.io/projected/4c444bca-6bc7-4d67-8e29-322ba9b044bd-kube-api-access-tfnmw\") pod \"maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz\" (UID: \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"
Sep 07 22:55:53.820930 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:53.820796    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c444bca-6bc7-4d67-8e29-322ba9b044bd-metrics-certs\") pod \"maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz\" (UID: \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"
Sep 07 22:55:53.921741 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:53.921648    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c444bca-6bc7-4d67-8e29-322ba9b044bd-metrics-certs\") pod \"maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz\" (UID: \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"
Sep 07 22:55:53.921907 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:53.921742    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c444bca-6bc7-4d67-8e29-322ba9b044bd-openshift-service-ca\") pod \"maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz\" (UID: \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"
Sep 07 22:55:53.921907 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:53.921767    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c444bca-6bc7-4d67-8e29-322ba9b044bd-maas-api-tls\") pod \"maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz\" (UID: \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"
Sep 07 22:55:53.921907 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:53.921811    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfnmw\" (UniqueName: \"kubernetes.io/projected/4c444bca-6bc7-4d67-8e29-322ba9b044bd-kube-api-access-tfnmw\") pod \"maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz\" (UID: \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"
Sep 07 22:55:53.921907 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:55:53.921892    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-serving-cert: secret "maas-api-e2e-api-a-dd5b43-serving-cert" not found
Sep 07 22:55:53.922092 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:55:53.921971    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c444bca-6bc7-4d67-8e29-322ba9b044bd-maas-api-tls podName:4c444bca-6bc7-4d67-8e29-322ba9b044bd nodeName:}" failed. No retries permitted until 2026-09-07 22:55:54.421949687 +0000 UTC m=+1437.478782642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4c444bca-6bc7-4d67-8e29-322ba9b044bd-maas-api-tls") pod "maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz" (UID: "4c444bca-6bc7-4d67-8e29-322ba9b044bd") : secret "maas-api-e2e-api-a-dd5b43-serving-cert" not found
Sep 07 22:55:53.922504 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:53.922485    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c444bca-6bc7-4d67-8e29-322ba9b044bd-openshift-service-ca\") pod \"maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz\" (UID: \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"
Sep 07 22:55:53.924118 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:53.924097    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c444bca-6bc7-4d67-8e29-322ba9b044bd-metrics-certs\") pod \"maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz\" (UID: \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"
Sep 07 22:55:53.931746 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:53.931709    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfnmw\" (UniqueName: \"kubernetes.io/projected/4c444bca-6bc7-4d67-8e29-322ba9b044bd-kube-api-access-tfnmw\") pod \"maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz\" (UID: \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"
Sep 07 22:55:54.262405 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:54.262318    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk"]
Sep 07 22:55:54.266975 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:54.266943    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk"
Sep 07 22:55:54.269391 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:54.269365    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-dd5b43\""
Sep 07 22:55:54.269526 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:54.269375    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-dd5b43-dockercfg-wpnwr\""
Sep 07 22:55:54.276398 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:54.276147    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk"]
Sep 07 22:55:54.428527 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:54.428480    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c444bca-6bc7-4d67-8e29-322ba9b044bd-maas-api-tls\") pod \"maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz\" (UID: \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"
Sep 07 22:55:54.428718 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:54.428579    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjngb\" (UniqueName: \"kubernetes.io/projected/918d99ef-bdb4-4618-9b40-9a601b24020d-kube-api-access-fjngb\") pod \"payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk\" (UID: \"918d99ef-bdb4-4618-9b40-9a601b24020d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk"
Sep 07 22:55:54.428773 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:54.428732    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918d99ef-bdb4-4618-9b40-9a601b24020d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk\" (UID: \"918d99ef-bdb4-4618-9b40-9a601b24020d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk"
Sep 07 22:55:54.431109 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:54.431078    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c444bca-6bc7-4d67-8e29-322ba9b044bd-maas-api-tls\") pod \"maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz\" (UID: \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"
Sep 07 22:55:54.529449 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:54.529352    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918d99ef-bdb4-4618-9b40-9a601b24020d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk\" (UID: \"918d99ef-bdb4-4618-9b40-9a601b24020d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk"
Sep 07 22:55:54.529449 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:54.529426    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjngb\" (UniqueName: \"kubernetes.io/projected/918d99ef-bdb4-4618-9b40-9a601b24020d-kube-api-access-fjngb\") pod \"payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk\" (UID: \"918d99ef-bdb4-4618-9b40-9a601b24020d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk"
Sep 07 22:55:54.530089 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:54.530067    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918d99ef-bdb4-4618-9b40-9a601b24020d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk\" (UID: \"918d99ef-bdb4-4618-9b40-9a601b24020d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk"
Sep 07 22:55:54.536982 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:54.536951    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjngb\" (UniqueName: \"kubernetes.io/projected/918d99ef-bdb4-4618-9b40-9a601b24020d-kube-api-access-fjngb\") pod \"payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk\" (UID: \"918d99ef-bdb4-4618-9b40-9a601b24020d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk"
Sep 07 22:55:54.566007 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:54.565968    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"
Sep 07 22:55:54.580762 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:54.580733    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk"
Sep 07 22:55:54.852304 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:54.850617    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"]
Sep 07 22:55:54.854711 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:55:54.854680    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c444bca_6bc7_4d67_8e29_322ba9b044bd.slice/crio-6e27f396e415ef8eab467063aeea4c8ae3866851e95a4c375e4766ef6f1cb81d WatchSource:0}: Error finding container 6e27f396e415ef8eab467063aeea4c8ae3866851e95a4c375e4766ef6f1cb81d: Status 404 returned error can't find the container with id 6e27f396e415ef8eab467063aeea4c8ae3866851e95a4c375e4766ef6f1cb81d
Sep 07 22:55:54.872972 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:54.872944    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk"]
Sep 07 22:55:54.875551 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:55:54.875520    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918d99ef_bdb4_4618_9b40_9a601b24020d.slice/crio-7d849e9d67cc686778992cb7b52d07539e53b97d583e29b6a3fc9b3da83105d4 WatchSource:0}: Error finding container 7d849e9d67cc686778992cb7b52d07539e53b97d583e29b6a3fc9b3da83105d4: Status 404 returned error can't find the container with id 7d849e9d67cc686778992cb7b52d07539e53b97d583e29b6a3fc9b3da83105d4
Sep 07 22:55:55.833724 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:55.833679    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk" event={"ID":"918d99ef-bdb4-4618-9b40-9a601b24020d","Type":"ContainerStarted","Data":"5fbbd32257d52cd63933a0c89027a0f4c64067eef3721c803355d89cde932595"}
Sep 07 22:55:55.833918 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:55.833748    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk" event={"ID":"918d99ef-bdb4-4618-9b40-9a601b24020d","Type":"ContainerStarted","Data":"7d849e9d67cc686778992cb7b52d07539e53b97d583e29b6a3fc9b3da83105d4"}
Sep 07 22:55:55.833918 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:55.833877    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk"
Sep 07 22:55:55.835383 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:55.835345    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz" event={"ID":"4c444bca-6bc7-4d67-8e29-322ba9b044bd","Type":"ContainerStarted","Data":"32240dad05b0063771f47dee63949697936810d4ad8e7de65f2332bd1c21f0cb"}
Sep 07 22:55:55.835490 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:55.835407    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"
Sep 07 22:55:55.835490 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:55.835431    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz" event={"ID":"4c444bca-6bc7-4d67-8e29-322ba9b044bd","Type":"ContainerStarted","Data":"6e27f396e415ef8eab467063aeea4c8ae3866851e95a4c375e4766ef6f1cb81d"}
Sep 07 22:55:55.851548 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:55.851482    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk" podStartSLOduration=1.851460643 podStartE2EDuration="1.851460643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:55:55.849307905 +0000 UTC m=+1438.906140867" watchObservedRunningTime="2026-09-07 22:55:55.851460643 +0000 UTC m=+1438.908293605"
Sep 07 22:55:55.866817 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:55:55.866751    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz" podStartSLOduration=2.412657855 podStartE2EDuration="2.866735069s" totalImagesPullingTime="454.077214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:55:55.86564916 +0000 UTC m=+1438.922482158" watchObservedRunningTime="2026-09-07 22:55:55.866735069 +0000 UTC m=+1438.923568048"
Sep 07 22:56:01.846051 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:01.846019    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"
Sep 07 22:56:06.842772 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:06.842740    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk"
Sep 07 22:56:23.580450 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.580407    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x"]
Sep 07 22:56:23.585443 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.585423    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x"
Sep 07 22:56:23.588962 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.588936    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c7776f-dockercfg-dfwkh\""
Sep 07 22:56:23.589178 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.589156    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c7776f\""
Sep 07 22:56:23.595898 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.595874    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k"]
Sep 07 22:56:23.599493 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.599473    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k"
Sep 07 22:56:23.602343 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.602317    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x"]
Sep 07 22:56:23.630313 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.630277    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k"]
Sep 07 22:56:23.717266 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.717205    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3f9312-3ada-404e-97e2-addd7947d228-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x\" (UID: \"2c3f9312-3ada-404e-97e2-addd7947d228\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x"
Sep 07 22:56:23.717478 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.717283    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fc29fd-39ba-4396-a151-942c5da72480-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k\" (UID: \"f9fc29fd-39ba-4396-a151-942c5da72480\") " pod="openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k"
Sep 07 22:56:23.717478 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.717406    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpvxw\" (UniqueName: \"kubernetes.io/projected/f9fc29fd-39ba-4396-a151-942c5da72480-kube-api-access-lpvxw\") pod \"payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k\" (UID: \"f9fc29fd-39ba-4396-a151-942c5da72480\") " pod="openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k"
Sep 07 22:56:23.717478 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.717440    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f62qv\" (UniqueName: \"kubernetes.io/projected/2c3f9312-3ada-404e-97e2-addd7947d228-kube-api-access-f62qv\") pod \"payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x\" (UID: \"2c3f9312-3ada-404e-97e2-addd7947d228\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x"
Sep 07 22:56:23.818166 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.818120    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3f9312-3ada-404e-97e2-addd7947d228-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x\" (UID: \"2c3f9312-3ada-404e-97e2-addd7947d228\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x"
Sep 07 22:56:23.818166 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.818166    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fc29fd-39ba-4396-a151-942c5da72480-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k\" (UID: \"f9fc29fd-39ba-4396-a151-942c5da72480\") " pod="openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k"
Sep 07 22:56:23.818424 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.818212    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpvxw\" (UniqueName: \"kubernetes.io/projected/f9fc29fd-39ba-4396-a151-942c5da72480-kube-api-access-lpvxw\") pod \"payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k\" (UID: \"f9fc29fd-39ba-4396-a151-942c5da72480\") " pod="openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k"
Sep 07 22:56:23.818424 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.818228    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f62qv\" (UniqueName: \"kubernetes.io/projected/2c3f9312-3ada-404e-97e2-addd7947d228-kube-api-access-f62qv\") pod \"payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x\" (UID: \"2c3f9312-3ada-404e-97e2-addd7947d228\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x"
Sep 07 22:56:23.818938 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.818916    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fc29fd-39ba-4396-a151-942c5da72480-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k\" (UID: \"f9fc29fd-39ba-4396-a151-942c5da72480\") " pod="openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k"
Sep 07 22:56:23.818982 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.818952    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3f9312-3ada-404e-97e2-addd7947d228-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x\" (UID: \"2c3f9312-3ada-404e-97e2-addd7947d228\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x"
Sep 07 22:56:23.825774 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.825738    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpvxw\" (UniqueName: \"kubernetes.io/projected/f9fc29fd-39ba-4396-a151-942c5da72480-kube-api-access-lpvxw\") pod \"payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k\" (UID: \"f9fc29fd-39ba-4396-a151-942c5da72480\") " pod="openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k"
Sep 07 22:56:23.825984 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.825963    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f62qv\" (UniqueName: \"kubernetes.io/projected/2c3f9312-3ada-404e-97e2-addd7947d228-kube-api-access-f62qv\") pod \"payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x\" (UID: \"2c3f9312-3ada-404e-97e2-addd7947d228\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x"
Sep 07 22:56:23.896668 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.896593    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x"
Sep 07 22:56:23.911165 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:23.911106    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k"
Sep 07 22:56:24.173771 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:24.173738    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x"]
Sep 07 22:56:24.176447 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:56:24.176414    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3f9312_3ada_404e_97e2_addd7947d228.slice/crio-6414278f28a7d6c73e31995dcf9c2db1f65c4c08ef40b80dcd9761222f1add1d WatchSource:0}: Error finding container 6414278f28a7d6c73e31995dcf9c2db1f65c4c08ef40b80dcd9761222f1add1d: Status 404 returned error can't find the container with id 6414278f28a7d6c73e31995dcf9c2db1f65c4c08ef40b80dcd9761222f1add1d
Sep 07 22:56:24.201159 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:24.201126    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k"]
Sep 07 22:56:24.209450 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:56:24.209424    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fc29fd_39ba_4396_a151_942c5da72480.slice/crio-cacbda4dc409ffee74a6f150cf560af6de3e9edff51e7b1778e77981e706b888 WatchSource:0}: Error finding container cacbda4dc409ffee74a6f150cf560af6de3e9edff51e7b1778e77981e706b888: Status 404 returned error can't find the container with id cacbda4dc409ffee74a6f150cf560af6de3e9edff51e7b1778e77981e706b888
Sep 07 22:56:25.001925 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:25.001890    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x" event={"ID":"2c3f9312-3ada-404e-97e2-addd7947d228","Type":"ContainerStarted","Data":"81a891f3cad0f2d3f290f9559bb4f0623f68dabbaa6c145dc2149dabcbc69842"}
Sep 07 22:56:25.002407 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:25.001948    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x" event={"ID":"2c3f9312-3ada-404e-97e2-addd7947d228","Type":"ContainerStarted","Data":"6414278f28a7d6c73e31995dcf9c2db1f65c4c08ef40b80dcd9761222f1add1d"}
Sep 07 22:56:25.002407 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:25.001996    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x"
Sep 07 22:56:25.003761 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:25.003729    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k" event={"ID":"f9fc29fd-39ba-4396-a151-942c5da72480","Type":"ContainerStarted","Data":"322f55b6f33528d6e12708b9c7eb95dafad3b31f1d139fdd8f114752ce6a40e9"}
Sep 07 22:56:25.003886 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:25.003785    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k" event={"ID":"f9fc29fd-39ba-4396-a151-942c5da72480","Type":"ContainerStarted","Data":"cacbda4dc409ffee74a6f150cf560af6de3e9edff51e7b1778e77981e706b888"}
Sep 07 22:56:25.003886 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:25.003826    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k"
Sep 07 22:56:25.024681 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:25.024602    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x" podStartSLOduration=2.024581006 podStartE2EDuration="2.024581006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:56:25.023580307 +0000 UTC m=+1468.080413291" watchObservedRunningTime="2026-09-07 22:56:25.024581006 +0000 UTC m=+1468.081413963"
Sep 07 22:56:25.040667 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:25.040590    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k" podStartSLOduration=2.040557905 podStartE2EDuration="2.040557905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:56:25.038235291 +0000 UTC m=+1468.095068255" watchObservedRunningTime="2026-09-07 22:56:25.040557905 +0000 UTC m=+1468.097390863"
Sep 07 22:56:36.011463 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:36.011423    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x"
Sep 07 22:56:36.011993 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:36.011898    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k"
Sep 07 22:56:57.508466 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:57.508429    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/2.log"
Sep 07 22:56:57.510804 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:56:57.510777    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/2.log"
Sep 07 22:57:04.909225 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:04.909188    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"]
Sep 07 22:57:04.909685 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:04.909461    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz" podUID="4c444bca-6bc7-4d67-8e29-322ba9b044bd" containerName="maas-api" containerID="cri-o://32240dad05b0063771f47dee63949697936810d4ad8e7de65f2332bd1c21f0cb" gracePeriod=30
Sep 07 22:57:04.972352 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:04.972313    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk"]
Sep 07 22:57:04.972621 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:04.972556    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk" podUID="918d99ef-bdb4-4618-9b40-9a601b24020d" containerName="payload-pre-processing" containerID="cri-o://5fbbd32257d52cd63933a0c89027a0f4c64067eef3721c803355d89cde932595" gracePeriod=30
Sep 07 22:57:05.231862 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.231831    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"
Sep 07 22:57:05.232599 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.232562    2567 generic.go:356] "Generic (PLEG): container finished" podID="918d99ef-bdb4-4618-9b40-9a601b24020d" containerID="5fbbd32257d52cd63933a0c89027a0f4c64067eef3721c803355d89cde932595" exitCode=0
Sep 07 22:57:05.232733 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.232623    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk" event={"ID":"918d99ef-bdb4-4618-9b40-9a601b24020d","Type":"ContainerDied","Data":"5fbbd32257d52cd63933a0c89027a0f4c64067eef3721c803355d89cde932595"}
Sep 07 22:57:05.234075 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.234049    2567 generic.go:356] "Generic (PLEG): container finished" podID="4c444bca-6bc7-4d67-8e29-322ba9b044bd" containerID="32240dad05b0063771f47dee63949697936810d4ad8e7de65f2332bd1c21f0cb" exitCode=0
Sep 07 22:57:05.234180 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.234079    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"
Sep 07 22:57:05.234180 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.234126    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz" event={"ID":"4c444bca-6bc7-4d67-8e29-322ba9b044bd","Type":"ContainerDied","Data":"32240dad05b0063771f47dee63949697936810d4ad8e7de65f2332bd1c21f0cb"}
Sep 07 22:57:05.234323 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.234191    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz" event={"ID":"4c444bca-6bc7-4d67-8e29-322ba9b044bd","Type":"ContainerDied","Data":"6e27f396e415ef8eab467063aeea4c8ae3866851e95a4c375e4766ef6f1cb81d"}
Sep 07 22:57:05.234323 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.234225    2567 scope.go:117] "RemoveContainer" containerID="32240dad05b0063771f47dee63949697936810d4ad8e7de65f2332bd1c21f0cb"
Sep 07 22:57:05.244625 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.244600    2567 scope.go:117] "RemoveContainer" containerID="32240dad05b0063771f47dee63949697936810d4ad8e7de65f2332bd1c21f0cb"
Sep 07 22:57:05.245034 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:57:05.245007    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32240dad05b0063771f47dee63949697936810d4ad8e7de65f2332bd1c21f0cb\": container with ID starting with 32240dad05b0063771f47dee63949697936810d4ad8e7de65f2332bd1c21f0cb not found: ID does not exist" containerID="32240dad05b0063771f47dee63949697936810d4ad8e7de65f2332bd1c21f0cb"
Sep 07 22:57:05.245157 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.245041    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32240dad05b0063771f47dee63949697936810d4ad8e7de65f2332bd1c21f0cb"} err="failed to get container status \"32240dad05b0063771f47dee63949697936810d4ad8e7de65f2332bd1c21f0cb\": rpc error: code = NotFound desc = could not find container \"32240dad05b0063771f47dee63949697936810d4ad8e7de65f2332bd1c21f0cb\": container with ID starting with 32240dad05b0063771f47dee63949697936810d4ad8e7de65f2332bd1c21f0cb not found: ID does not exist"
Sep 07 22:57:05.268556 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.268530    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk"
Sep 07 22:57:05.337376 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.337331    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c444bca-6bc7-4d67-8e29-322ba9b044bd-openshift-service-ca\") pod \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\" (UID: \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\") "
Sep 07 22:57:05.337376 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.337378    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfnmw\" (UniqueName: \"kubernetes.io/projected/4c444bca-6bc7-4d67-8e29-322ba9b044bd-kube-api-access-tfnmw\") pod \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\" (UID: \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\") "
Sep 07 22:57:05.337596 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.337509    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c444bca-6bc7-4d67-8e29-322ba9b044bd-maas-api-tls\") pod \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\" (UID: \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\") "
Sep 07 22:57:05.337596 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.337591    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c444bca-6bc7-4d67-8e29-322ba9b044bd-metrics-certs\") pod \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\" (UID: \"4c444bca-6bc7-4d67-8e29-322ba9b044bd\") "
Sep 07 22:57:05.337791 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.337764    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c444bca-6bc7-4d67-8e29-322ba9b044bd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4c444bca-6bc7-4d67-8e29-322ba9b044bd" (UID: "4c444bca-6bc7-4d67-8e29-322ba9b044bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:57:05.337915 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.337900    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c444bca-6bc7-4d67-8e29-322ba9b044bd-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:57:05.339703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.339671    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c444bca-6bc7-4d67-8e29-322ba9b044bd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4c444bca-6bc7-4d67-8e29-322ba9b044bd" (UID: "4c444bca-6bc7-4d67-8e29-322ba9b044bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:57:05.339703 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.339678    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c444bca-6bc7-4d67-8e29-322ba9b044bd-kube-api-access-tfnmw" (OuterVolumeSpecName: "kube-api-access-tfnmw") pod "4c444bca-6bc7-4d67-8e29-322ba9b044bd" (UID: "4c444bca-6bc7-4d67-8e29-322ba9b044bd"). InnerVolumeSpecName "kube-api-access-tfnmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:57:05.339837 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.339699    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c444bca-6bc7-4d67-8e29-322ba9b044bd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4c444bca-6bc7-4d67-8e29-322ba9b044bd" (UID: "4c444bca-6bc7-4d67-8e29-322ba9b044bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:57:05.438767 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.438592    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918d99ef-bdb4-4618-9b40-9a601b24020d-plugins-config-volume\") pod \"918d99ef-bdb4-4618-9b40-9a601b24020d\" (UID: \"918d99ef-bdb4-4618-9b40-9a601b24020d\") "
Sep 07 22:57:05.438767 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.438689    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjngb\" (UniqueName: \"kubernetes.io/projected/918d99ef-bdb4-4618-9b40-9a601b24020d-kube-api-access-fjngb\") pod \"918d99ef-bdb4-4618-9b40-9a601b24020d\" (UID: \"918d99ef-bdb4-4618-9b40-9a601b24020d\") "
Sep 07 22:57:05.439137 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.439117    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c444bca-6bc7-4d67-8e29-322ba9b044bd-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:57:05.439137 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.439137    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c444bca-6bc7-4d67-8e29-322ba9b044bd-metrics-certs\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:57:05.439293 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.439148    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfnmw\" (UniqueName: \"kubernetes.io/projected/4c444bca-6bc7-4d67-8e29-322ba9b044bd-kube-api-access-tfnmw\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:57:05.439560 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.439522    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918d99ef-bdb4-4618-9b40-9a601b24020d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "918d99ef-bdb4-4618-9b40-9a601b24020d" (UID: "918d99ef-bdb4-4618-9b40-9a601b24020d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:57:05.441184 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.441165    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918d99ef-bdb4-4618-9b40-9a601b24020d-kube-api-access-fjngb" (OuterVolumeSpecName: "kube-api-access-fjngb") pod "918d99ef-bdb4-4618-9b40-9a601b24020d" (UID: "918d99ef-bdb4-4618-9b40-9a601b24020d"). InnerVolumeSpecName "kube-api-access-fjngb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:57:05.542693 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.541117    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918d99ef-bdb4-4618-9b40-9a601b24020d-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:57:05.542693 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.541149    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjngb\" (UniqueName: \"kubernetes.io/projected/918d99ef-bdb4-4618-9b40-9a601b24020d-kube-api-access-fjngb\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:57:05.550907 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.550858    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"]
Sep 07 22:57:05.552885 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:05.552856    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd5b43-67d5d74476-pnrhz"]
Sep 07 22:57:06.242754 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:06.242670    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk"
Sep 07 22:57:06.242754 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:06.242683    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk" event={"ID":"918d99ef-bdb4-4618-9b40-9a601b24020d","Type":"ContainerDied","Data":"7d849e9d67cc686778992cb7b52d07539e53b97d583e29b6a3fc9b3da83105d4"}
Sep 07 22:57:06.243299 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:06.242778    2567 scope.go:117] "RemoveContainer" containerID="5fbbd32257d52cd63933a0c89027a0f4c64067eef3721c803355d89cde932595"
Sep 07 22:57:06.257357 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:06.257319    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk"]
Sep 07 22:57:06.260750 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:06.260717    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd5b43-78df975cbb-j9ptk"]
Sep 07 22:57:07.465698 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:07.465662    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c444bca-6bc7-4d67-8e29-322ba9b044bd" path="/var/lib/kubelet/pods/4c444bca-6bc7-4d67-8e29-322ba9b044bd/volumes"
Sep 07 22:57:07.466158 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:07.466024    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="918d99ef-bdb4-4618-9b40-9a601b24020d" path="/var/lib/kubelet/pods/918d99ef-bdb4-4618-9b40-9a601b24020d/volumes"
Sep 07 22:57:27.274316 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:27.274187    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k"]
Sep 07 22:57:27.274867 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:27.274462    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k" podUID="f9fc29fd-39ba-4396-a151-942c5da72480" containerName="payload-processing" containerID="cri-o://322f55b6f33528d6e12708b9c7eb95dafad3b31f1d139fdd8f114752ce6a40e9" gracePeriod=30
Sep 07 22:57:27.280930 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:27.280898    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x"]
Sep 07 22:57:27.281186 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:27.281144    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x" podUID="2c3f9312-3ada-404e-97e2-addd7947d228" containerName="payload-pre-processing" containerID="cri-o://81a891f3cad0f2d3f290f9559bb4f0623f68dabbaa6c145dc2149dabcbc69842" gracePeriod=30
Sep 07 22:57:27.566055 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:27.566025    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k"
Sep 07 22:57:27.568947 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:27.568926    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x"
Sep 07 22:57:27.668797 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:27.668761    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fc29fd-39ba-4396-a151-942c5da72480-plugins-config-volume\") pod \"f9fc29fd-39ba-4396-a151-942c5da72480\" (UID: \"f9fc29fd-39ba-4396-a151-942c5da72480\") "
Sep 07 22:57:27.668797 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:27.668803    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpvxw\" (UniqueName: \"kubernetes.io/projected/f9fc29fd-39ba-4396-a151-942c5da72480-kube-api-access-lpvxw\") pod \"f9fc29fd-39ba-4396-a151-942c5da72480\" (UID: \"f9fc29fd-39ba-4396-a151-942c5da72480\") "
Sep 07 22:57:27.669023 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:27.668939    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3f9312-3ada-404e-97e2-addd7947d228-plugins-config-volume\") pod \"2c3f9312-3ada-404e-97e2-addd7947d228\" (UID: \"2c3f9312-3ada-404e-97e2-addd7947d228\") "
Sep 07 22:57:27.669023 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:27.668993    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f62qv\" (UniqueName: \"kubernetes.io/projected/2c3f9312-3ada-404e-97e2-addd7947d228-kube-api-access-f62qv\") pod \"2c3f9312-3ada-404e-97e2-addd7947d228\" (UID: \"2c3f9312-3ada-404e-97e2-addd7947d228\") "
Sep 07 22:57:27.669190 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:27.669162    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fc29fd-39ba-4396-a151-942c5da72480-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9fc29fd-39ba-4396-a151-942c5da72480" (UID: "f9fc29fd-39ba-4396-a151-942c5da72480"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:57:27.669395 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:27.669374    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fc29fd-39ba-4396-a151-942c5da72480-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:57:27.669470 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:27.669386    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3f9312-3ada-404e-97e2-addd7947d228-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c3f9312-3ada-404e-97e2-addd7947d228" (UID: "2c3f9312-3ada-404e-97e2-addd7947d228"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:57:27.671097 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:27.671071    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3f9312-3ada-404e-97e2-addd7947d228-kube-api-access-f62qv" (OuterVolumeSpecName: "kube-api-access-f62qv") pod "2c3f9312-3ada-404e-97e2-addd7947d228" (UID: "2c3f9312-3ada-404e-97e2-addd7947d228"). InnerVolumeSpecName "kube-api-access-f62qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:57:27.671158 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:27.671083    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9fc29fd-39ba-4396-a151-942c5da72480-kube-api-access-lpvxw" (OuterVolumeSpecName: "kube-api-access-lpvxw") pod "f9fc29fd-39ba-4396-a151-942c5da72480" (UID: "f9fc29fd-39ba-4396-a151-942c5da72480"). InnerVolumeSpecName "kube-api-access-lpvxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:57:27.770035 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:27.769991    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpvxw\" (UniqueName: \"kubernetes.io/projected/f9fc29fd-39ba-4396-a151-942c5da72480-kube-api-access-lpvxw\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:57:27.770035 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:27.770028    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3f9312-3ada-404e-97e2-addd7947d228-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:57:27.770035 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:27.770040    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f62qv\" (UniqueName: \"kubernetes.io/projected/2c3f9312-3ada-404e-97e2-addd7947d228-kube-api-access-f62qv\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:57:28.359797 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:28.359758    2567 generic.go:356] "Generic (PLEG): container finished" podID="f9fc29fd-39ba-4396-a151-942c5da72480" containerID="322f55b6f33528d6e12708b9c7eb95dafad3b31f1d139fdd8f114752ce6a40e9" exitCode=0
Sep 07 22:57:28.360291 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:28.359803    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k"
Sep 07 22:57:28.360291 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:28.359842    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k" event={"ID":"f9fc29fd-39ba-4396-a151-942c5da72480","Type":"ContainerDied","Data":"322f55b6f33528d6e12708b9c7eb95dafad3b31f1d139fdd8f114752ce6a40e9"}
Sep 07 22:57:28.360291 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:28.359894    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k" event={"ID":"f9fc29fd-39ba-4396-a151-942c5da72480","Type":"ContainerDied","Data":"cacbda4dc409ffee74a6f150cf560af6de3e9edff51e7b1778e77981e706b888"}
Sep 07 22:57:28.360291 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:28.359918    2567 scope.go:117] "RemoveContainer" containerID="322f55b6f33528d6e12708b9c7eb95dafad3b31f1d139fdd8f114752ce6a40e9"
Sep 07 22:57:28.361425 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:28.361404    2567 generic.go:356] "Generic (PLEG): container finished" podID="2c3f9312-3ada-404e-97e2-addd7947d228" containerID="81a891f3cad0f2d3f290f9559bb4f0623f68dabbaa6c145dc2149dabcbc69842" exitCode=0
Sep 07 22:57:28.361488 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:28.361431    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x" event={"ID":"2c3f9312-3ada-404e-97e2-addd7947d228","Type":"ContainerDied","Data":"81a891f3cad0f2d3f290f9559bb4f0623f68dabbaa6c145dc2149dabcbc69842"}
Sep 07 22:57:28.361488 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:28.361468    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x" event={"ID":"2c3f9312-3ada-404e-97e2-addd7947d228","Type":"ContainerDied","Data":"6414278f28a7d6c73e31995dcf9c2db1f65c4c08ef40b80dcd9761222f1add1d"}
Sep 07 22:57:28.361558 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:28.361517    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x"
Sep 07 22:57:28.369759 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:28.369733    2567 scope.go:117] "RemoveContainer" containerID="322f55b6f33528d6e12708b9c7eb95dafad3b31f1d139fdd8f114752ce6a40e9"
Sep 07 22:57:28.370148 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:57:28.370124    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322f55b6f33528d6e12708b9c7eb95dafad3b31f1d139fdd8f114752ce6a40e9\": container with ID starting with 322f55b6f33528d6e12708b9c7eb95dafad3b31f1d139fdd8f114752ce6a40e9 not found: ID does not exist" containerID="322f55b6f33528d6e12708b9c7eb95dafad3b31f1d139fdd8f114752ce6a40e9"
Sep 07 22:57:28.370266 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:28.370159    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322f55b6f33528d6e12708b9c7eb95dafad3b31f1d139fdd8f114752ce6a40e9"} err="failed to get container status \"322f55b6f33528d6e12708b9c7eb95dafad3b31f1d139fdd8f114752ce6a40e9\": rpc error: code = NotFound desc = could not find container \"322f55b6f33528d6e12708b9c7eb95dafad3b31f1d139fdd8f114752ce6a40e9\": container with ID starting with 322f55b6f33528d6e12708b9c7eb95dafad3b31f1d139fdd8f114752ce6a40e9 not found: ID does not exist"
Sep 07 22:57:28.370266 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:28.370177    2567 scope.go:117] "RemoveContainer" containerID="81a891f3cad0f2d3f290f9559bb4f0623f68dabbaa6c145dc2149dabcbc69842"
Sep 07 22:57:28.379493 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:28.379472    2567 scope.go:117] "RemoveContainer" containerID="81a891f3cad0f2d3f290f9559bb4f0623f68dabbaa6c145dc2149dabcbc69842"
Sep 07 22:57:28.379813 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:57:28.379793    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a891f3cad0f2d3f290f9559bb4f0623f68dabbaa6c145dc2149dabcbc69842\": container with ID starting with 81a891f3cad0f2d3f290f9559bb4f0623f68dabbaa6c145dc2149dabcbc69842 not found: ID does not exist" containerID="81a891f3cad0f2d3f290f9559bb4f0623f68dabbaa6c145dc2149dabcbc69842"
Sep 07 22:57:28.379880 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:28.379826    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a891f3cad0f2d3f290f9559bb4f0623f68dabbaa6c145dc2149dabcbc69842"} err="failed to get container status \"81a891f3cad0f2d3f290f9559bb4f0623f68dabbaa6c145dc2149dabcbc69842\": rpc error: code = NotFound desc = could not find container \"81a891f3cad0f2d3f290f9559bb4f0623f68dabbaa6c145dc2149dabcbc69842\": container with ID starting with 81a891f3cad0f2d3f290f9559bb4f0623f68dabbaa6c145dc2149dabcbc69842 not found: ID does not exist"
Sep 07 22:57:28.385365 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:28.385227    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k"]
Sep 07 22:57:28.392107 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:28.392080    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c7776f-7dd8bb74c7-5mz5k"]
Sep 07 22:57:28.401683 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:28.401653    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x"]
Sep 07 22:57:28.405109 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:28.405079    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c7776f-77b6bd6d9d-w7z9x"]
Sep 07 22:57:29.466461 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:29.466431    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c3f9312-3ada-404e-97e2-addd7947d228" path="/var/lib/kubelet/pods/2c3f9312-3ada-404e-97e2-addd7947d228/volumes"
Sep 07 22:57:29.466843 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:29.466828    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9fc29fd-39ba-4396-a151-942c5da72480" path="/var/lib/kubelet/pods/f9fc29fd-39ba-4396-a151-942c5da72480/volumes"
Sep 07 22:57:44.528876 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.528839    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"]
Sep 07 22:57:44.531080 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.529073    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x" podUID="564bcfd9-41b6-4ee4-b553-44bccb29a8cd" containerName="maas-api" containerID="cri-o://9da03a516473c037839062a98bef980f1a79354aa0a74c06747fde683268ecf0" gracePeriod=30
Sep 07 22:57:44.573685 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.573653    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48"]
Sep 07 22:57:44.573886 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.573862    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48" podUID="11efd1f5-0a20-4c7a-841d-e87b11f492f7" containerName="payload-processing" containerID="cri-o://21895f16a1067adf6469b651459d2df413c0ee90ff1066c0cada91bb133bf270" gracePeriod=30
Sep 07 22:57:44.845422 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.845392    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"
Sep 07 22:57:44.848427 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.848399    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48"
Sep 07 22:57:44.940329 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.940291    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11efd1f5-0a20-4c7a-841d-e87b11f492f7-plugins-config-volume\") pod \"11efd1f5-0a20-4c7a-841d-e87b11f492f7\" (UID: \"11efd1f5-0a20-4c7a-841d-e87b11f492f7\") "
Sep 07 22:57:44.940540 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.940410    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-openshift-service-ca\") pod \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\" (UID: \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\") "
Sep 07 22:57:44.940540 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.940497    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c2f6\" (UniqueName: \"kubernetes.io/projected/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-kube-api-access-9c2f6\") pod \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\" (UID: \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\") "
Sep 07 22:57:44.940679 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.940563    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-metrics-certs\") pod \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\" (UID: \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\") "
Sep 07 22:57:44.940679 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.940619    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-maas-api-tls\") pod \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\" (UID: \"564bcfd9-41b6-4ee4-b553-44bccb29a8cd\") "
Sep 07 22:57:44.940785 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.940681    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knzcr\" (UniqueName: \"kubernetes.io/projected/11efd1f5-0a20-4c7a-841d-e87b11f492f7-kube-api-access-knzcr\") pod \"11efd1f5-0a20-4c7a-841d-e87b11f492f7\" (UID: \"11efd1f5-0a20-4c7a-841d-e87b11f492f7\") "
Sep 07 22:57:44.940785 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.940715    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11efd1f5-0a20-4c7a-841d-e87b11f492f7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11efd1f5-0a20-4c7a-841d-e87b11f492f7" (UID: "11efd1f5-0a20-4c7a-841d-e87b11f492f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:57:44.940785 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.940762    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "564bcfd9-41b6-4ee4-b553-44bccb29a8cd" (UID: "564bcfd9-41b6-4ee4-b553-44bccb29a8cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:57:44.941043 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.941019    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11efd1f5-0a20-4c7a-841d-e87b11f492f7-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:57:44.941160 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.941051    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:57:44.942723 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.942697    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "564bcfd9-41b6-4ee4-b553-44bccb29a8cd" (UID: "564bcfd9-41b6-4ee4-b553-44bccb29a8cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:57:44.942893 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.942871    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11efd1f5-0a20-4c7a-841d-e87b11f492f7-kube-api-access-knzcr" (OuterVolumeSpecName: "kube-api-access-knzcr") pod "11efd1f5-0a20-4c7a-841d-e87b11f492f7" (UID: "11efd1f5-0a20-4c7a-841d-e87b11f492f7"). InnerVolumeSpecName "kube-api-access-knzcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:57:44.943073 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.943044    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "564bcfd9-41b6-4ee4-b553-44bccb29a8cd" (UID: "564bcfd9-41b6-4ee4-b553-44bccb29a8cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:57:44.943160 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:44.943100    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-kube-api-access-9c2f6" (OuterVolumeSpecName: "kube-api-access-9c2f6") pod "564bcfd9-41b6-4ee4-b553-44bccb29a8cd" (UID: "564bcfd9-41b6-4ee4-b553-44bccb29a8cd"). InnerVolumeSpecName "kube-api-access-9c2f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:57:45.042531 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.042498    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c2f6\" (UniqueName: \"kubernetes.io/projected/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-kube-api-access-9c2f6\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:57:45.042531 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.042527    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-metrics-certs\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:57:45.042531 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.042539    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/564bcfd9-41b6-4ee4-b553-44bccb29a8cd-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:57:45.042770 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.042548    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knzcr\" (UniqueName: \"kubernetes.io/projected/11efd1f5-0a20-4c7a-841d-e87b11f492f7-kube-api-access-knzcr\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:57:45.455098 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.455062    2567 generic.go:356] "Generic (PLEG): container finished" podID="564bcfd9-41b6-4ee4-b553-44bccb29a8cd" containerID="9da03a516473c037839062a98bef980f1a79354aa0a74c06747fde683268ecf0" exitCode=0
Sep 07 22:57:45.455294 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.455106    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"
Sep 07 22:57:45.455294 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.455146    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x" event={"ID":"564bcfd9-41b6-4ee4-b553-44bccb29a8cd","Type":"ContainerDied","Data":"9da03a516473c037839062a98bef980f1a79354aa0a74c06747fde683268ecf0"}
Sep 07 22:57:45.455294 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.455237    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x" event={"ID":"564bcfd9-41b6-4ee4-b553-44bccb29a8cd","Type":"ContainerDied","Data":"706738deb8b5a0803dac3f1083bbd013ec6700ffb6e74ade55bfdfdb81734fc9"}
Sep 07 22:57:45.455457 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.455294    2567 scope.go:117] "RemoveContainer" containerID="9da03a516473c037839062a98bef980f1a79354aa0a74c06747fde683268ecf0"
Sep 07 22:57:45.457040 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.457010    2567 generic.go:356] "Generic (PLEG): container finished" podID="11efd1f5-0a20-4c7a-841d-e87b11f492f7" containerID="21895f16a1067adf6469b651459d2df413c0ee90ff1066c0cada91bb133bf270" exitCode=0
Sep 07 22:57:45.457297 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.457083    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48"
Sep 07 22:57:45.457297 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.457124    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48" event={"ID":"11efd1f5-0a20-4c7a-841d-e87b11f492f7","Type":"ContainerDied","Data":"21895f16a1067adf6469b651459d2df413c0ee90ff1066c0cada91bb133bf270"}
Sep 07 22:57:45.457297 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.457163    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48" event={"ID":"11efd1f5-0a20-4c7a-841d-e87b11f492f7","Type":"ContainerDied","Data":"2841a78850b6fa9edf5227e68c46238d3d54a145a9b3418aabcf7a57d4a6e064"}
Sep 07 22:57:45.468011 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.467987    2567 scope.go:117] "RemoveContainer" containerID="9da03a516473c037839062a98bef980f1a79354aa0a74c06747fde683268ecf0"
Sep 07 22:57:45.468404 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:57:45.468378    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9da03a516473c037839062a98bef980f1a79354aa0a74c06747fde683268ecf0\": container with ID starting with 9da03a516473c037839062a98bef980f1a79354aa0a74c06747fde683268ecf0 not found: ID does not exist" containerID="9da03a516473c037839062a98bef980f1a79354aa0a74c06747fde683268ecf0"
Sep 07 22:57:45.468497 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.468411    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9da03a516473c037839062a98bef980f1a79354aa0a74c06747fde683268ecf0"} err="failed to get container status \"9da03a516473c037839062a98bef980f1a79354aa0a74c06747fde683268ecf0\": rpc error: code = NotFound desc = could not find container \"9da03a516473c037839062a98bef980f1a79354aa0a74c06747fde683268ecf0\": container with ID starting with 9da03a516473c037839062a98bef980f1a79354aa0a74c06747fde683268ecf0 not found: ID does not exist"
Sep 07 22:57:45.468497 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.468430    2567 scope.go:117] "RemoveContainer" containerID="21895f16a1067adf6469b651459d2df413c0ee90ff1066c0cada91bb133bf270"
Sep 07 22:57:45.477462 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.477442    2567 scope.go:117] "RemoveContainer" containerID="21895f16a1067adf6469b651459d2df413c0ee90ff1066c0cada91bb133bf270"
Sep 07 22:57:45.477771 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:57:45.477752    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21895f16a1067adf6469b651459d2df413c0ee90ff1066c0cada91bb133bf270\": container with ID starting with 21895f16a1067adf6469b651459d2df413c0ee90ff1066c0cada91bb133bf270 not found: ID does not exist" containerID="21895f16a1067adf6469b651459d2df413c0ee90ff1066c0cada91bb133bf270"
Sep 07 22:57:45.477819 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.477779    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21895f16a1067adf6469b651459d2df413c0ee90ff1066c0cada91bb133bf270"} err="failed to get container status \"21895f16a1067adf6469b651459d2df413c0ee90ff1066c0cada91bb133bf270\": rpc error: code = NotFound desc = could not find container \"21895f16a1067adf6469b651459d2df413c0ee90ff1066c0cada91bb133bf270\": container with ID starting with 21895f16a1067adf6469b651459d2df413c0ee90ff1066c0cada91bb133bf270 not found: ID does not exist"
Sep 07 22:57:45.482892 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.482867    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"]
Sep 07 22:57:45.486863 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.486835    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-82409b-679674fbd4-rnv4x"]
Sep 07 22:57:45.497343 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.497299    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48"]
Sep 07 22:57:45.499150 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:45.499128    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-82409b-6db8ddbcd7-dwq48"]
Sep 07 22:57:47.468517 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:47.468483    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11efd1f5-0a20-4c7a-841d-e87b11f492f7" path="/var/lib/kubelet/pods/11efd1f5-0a20-4c7a-841d-e87b11f492f7/volumes"
Sep 07 22:57:47.468992 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:57:47.468874    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="564bcfd9-41b6-4ee4-b553-44bccb29a8cd" path="/var/lib/kubelet/pods/564bcfd9-41b6-4ee4-b553-44bccb29a8cd/volumes"
Sep 07 22:58:01.779383 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:01.779339    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6"]
Sep 07 22:58:01.779910 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:01.779601    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6" podUID="76aba709-84f3-4332-b14e-5c8f75f5ae66" containerName="payload-processing" containerID="cri-o://793119d8e98f4dfa41e777526c93871af559ed48fe89c4043cd97c5a1b642f39" gracePeriod=30
Sep 07 22:58:01.787667 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:01.787617    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk"]
Sep 07 22:58:01.788281 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:01.788067    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk" podUID="5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde" containerName="payload-pre-processing" containerID="cri-o://fb2aabc868b60e2c9f8fd9dde54120fd97a3720389fd03da59fbd1220719ac34" gracePeriod=30
Sep 07 22:58:02.083283 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.083239    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6"
Sep 07 22:58:02.086151 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.086125    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk"
Sep 07 22:58:02.202801 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.202756    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zrmd\" (UniqueName: \"kubernetes.io/projected/76aba709-84f3-4332-b14e-5c8f75f5ae66-kube-api-access-9zrmd\") pod \"76aba709-84f3-4332-b14e-5c8f75f5ae66\" (UID: \"76aba709-84f3-4332-b14e-5c8f75f5ae66\") "
Sep 07 22:58:02.202801 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.202800    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s67hd\" (UniqueName: \"kubernetes.io/projected/5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde-kube-api-access-s67hd\") pod \"5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde\" (UID: \"5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde\") "
Sep 07 22:58:02.203071 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.202854    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde-plugins-config-volume\") pod \"5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde\" (UID: \"5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde\") "
Sep 07 22:58:02.203071 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.202883    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76aba709-84f3-4332-b14e-5c8f75f5ae66-plugins-config-volume\") pod \"76aba709-84f3-4332-b14e-5c8f75f5ae66\" (UID: \"76aba709-84f3-4332-b14e-5c8f75f5ae66\") "
Sep 07 22:58:02.203358 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.203326    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde" (UID: "5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:58:02.203448 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.203368    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76aba709-84f3-4332-b14e-5c8f75f5ae66-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "76aba709-84f3-4332-b14e-5c8f75f5ae66" (UID: "76aba709-84f3-4332-b14e-5c8f75f5ae66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:58:02.204930 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.204909    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76aba709-84f3-4332-b14e-5c8f75f5ae66-kube-api-access-9zrmd" (OuterVolumeSpecName: "kube-api-access-9zrmd") pod "76aba709-84f3-4332-b14e-5c8f75f5ae66" (UID: "76aba709-84f3-4332-b14e-5c8f75f5ae66"). InnerVolumeSpecName "kube-api-access-9zrmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:58:02.205009 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.204963    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde-kube-api-access-s67hd" (OuterVolumeSpecName: "kube-api-access-s67hd") pod "5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde" (UID: "5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde"). InnerVolumeSpecName "kube-api-access-s67hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:58:02.303918 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.303818    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:58:02.303918 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.303855    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76aba709-84f3-4332-b14e-5c8f75f5ae66-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:58:02.303918 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.303871    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zrmd\" (UniqueName: \"kubernetes.io/projected/76aba709-84f3-4332-b14e-5c8f75f5ae66-kube-api-access-9zrmd\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:58:02.303918 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.303889    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s67hd\" (UniqueName: \"kubernetes.io/projected/5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde-kube-api-access-s67hd\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 22:58:02.550832 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.550797    2567 generic.go:356] "Generic (PLEG): container finished" podID="5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde" containerID="fb2aabc868b60e2c9f8fd9dde54120fd97a3720389fd03da59fbd1220719ac34" exitCode=0
Sep 07 22:58:02.551033 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.550841    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk"
Sep 07 22:58:02.551033 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.550888    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk" event={"ID":"5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde","Type":"ContainerDied","Data":"fb2aabc868b60e2c9f8fd9dde54120fd97a3720389fd03da59fbd1220719ac34"}
Sep 07 22:58:02.551033 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.550951    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk" event={"ID":"5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde","Type":"ContainerDied","Data":"74989033581a8777a73e21abd5e4c27575de17e517422ec538644503c6b32aff"}
Sep 07 22:58:02.551033 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.550980    2567 scope.go:117] "RemoveContainer" containerID="fb2aabc868b60e2c9f8fd9dde54120fd97a3720389fd03da59fbd1220719ac34"
Sep 07 22:58:02.552392 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.552366    2567 generic.go:356] "Generic (PLEG): container finished" podID="76aba709-84f3-4332-b14e-5c8f75f5ae66" containerID="793119d8e98f4dfa41e777526c93871af559ed48fe89c4043cd97c5a1b642f39" exitCode=0
Sep 07 22:58:02.552547 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.552408    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6"
Sep 07 22:58:02.552547 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.552442    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6" event={"ID":"76aba709-84f3-4332-b14e-5c8f75f5ae66","Type":"ContainerDied","Data":"793119d8e98f4dfa41e777526c93871af559ed48fe89c4043cd97c5a1b642f39"}
Sep 07 22:58:02.552547 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.552480    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6" event={"ID":"76aba709-84f3-4332-b14e-5c8f75f5ae66","Type":"ContainerDied","Data":"781a08a319d0169d6de6589c4f77a77ca668ddfb2c181c9d1c9ff1492fca2387"}
Sep 07 22:58:02.561773 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.561749    2567 scope.go:117] "RemoveContainer" containerID="fb2aabc868b60e2c9f8fd9dde54120fd97a3720389fd03da59fbd1220719ac34"
Sep 07 22:58:02.562087 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:58:02.562069    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb2aabc868b60e2c9f8fd9dde54120fd97a3720389fd03da59fbd1220719ac34\": container with ID starting with fb2aabc868b60e2c9f8fd9dde54120fd97a3720389fd03da59fbd1220719ac34 not found: ID does not exist" containerID="fb2aabc868b60e2c9f8fd9dde54120fd97a3720389fd03da59fbd1220719ac34"
Sep 07 22:58:02.562148 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.562095    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2aabc868b60e2c9f8fd9dde54120fd97a3720389fd03da59fbd1220719ac34"} err="failed to get container status \"fb2aabc868b60e2c9f8fd9dde54120fd97a3720389fd03da59fbd1220719ac34\": rpc error: code = NotFound desc = could not find container \"fb2aabc868b60e2c9f8fd9dde54120fd97a3720389fd03da59fbd1220719ac34\": container with ID starting with fb2aabc868b60e2c9f8fd9dde54120fd97a3720389fd03da59fbd1220719ac34 not found: ID does not exist"
Sep 07 22:58:02.562148 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.562112    2567 scope.go:117] "RemoveContainer" containerID="793119d8e98f4dfa41e777526c93871af559ed48fe89c4043cd97c5a1b642f39"
Sep 07 22:58:02.571737 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.571708    2567 scope.go:117] "RemoveContainer" containerID="793119d8e98f4dfa41e777526c93871af559ed48fe89c4043cd97c5a1b642f39"
Sep 07 22:58:02.572118 ip-10-0-138-99 kubenswrapper[2567]: E0907 22:58:02.572089    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"793119d8e98f4dfa41e777526c93871af559ed48fe89c4043cd97c5a1b642f39\": container with ID starting with 793119d8e98f4dfa41e777526c93871af559ed48fe89c4043cd97c5a1b642f39 not found: ID does not exist" containerID="793119d8e98f4dfa41e777526c93871af559ed48fe89c4043cd97c5a1b642f39"
Sep 07 22:58:02.572234 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.572129    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"793119d8e98f4dfa41e777526c93871af559ed48fe89c4043cd97c5a1b642f39"} err="failed to get container status \"793119d8e98f4dfa41e777526c93871af559ed48fe89c4043cd97c5a1b642f39\": rpc error: code = NotFound desc = could not find container \"793119d8e98f4dfa41e777526c93871af559ed48fe89c4043cd97c5a1b642f39\": container with ID starting with 793119d8e98f4dfa41e777526c93871af559ed48fe89c4043cd97c5a1b642f39 not found: ID does not exist"
Sep 07 22:58:02.576523 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.576494    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk"]
Sep 07 22:58:02.582013 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.581982    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5483b1-6cd6f6cdd6-ds5tk"]
Sep 07 22:58:02.590860 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.590824    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6"]
Sep 07 22:58:02.594404 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:02.594375    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5483b1-5bd5f7b77c-w8bz6"]
Sep 07 22:58:03.466895 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:03.466847    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde" path="/var/lib/kubelet/pods/5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde/volumes"
Sep 07 22:58:03.467437 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:03.467416    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76aba709-84f3-4332-b14e-5c8f75f5ae66" path="/var/lib/kubelet/pods/76aba709-84f3-4332-b14e-5c8f75f5ae66/volumes"
Sep 07 22:58:06.510191 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.510155    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"]
Sep 07 22:58:06.510810 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.510791    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76aba709-84f3-4332-b14e-5c8f75f5ae66" containerName="payload-processing"
Sep 07 22:58:06.510860 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.510816    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="76aba709-84f3-4332-b14e-5c8f75f5ae66" containerName="payload-processing"
Sep 07 22:58:06.510860 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.510835    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="564bcfd9-41b6-4ee4-b553-44bccb29a8cd" containerName="maas-api"
Sep 07 22:58:06.510860 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.510844    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="564bcfd9-41b6-4ee4-b553-44bccb29a8cd" containerName="maas-api"
Sep 07 22:58:06.510860 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.510854    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde" containerName="payload-pre-processing"
Sep 07 22:58:06.510981 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.510863    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde" containerName="payload-pre-processing"
Sep 07 22:58:06.510981 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.510876    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11efd1f5-0a20-4c7a-841d-e87b11f492f7" containerName="payload-processing"
Sep 07 22:58:06.510981 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.510885    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="11efd1f5-0a20-4c7a-841d-e87b11f492f7" containerName="payload-processing"
Sep 07 22:58:06.510981 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.510897    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="918d99ef-bdb4-4618-9b40-9a601b24020d" containerName="payload-pre-processing"
Sep 07 22:58:06.510981 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.510906    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="918d99ef-bdb4-4618-9b40-9a601b24020d" containerName="payload-pre-processing"
Sep 07 22:58:06.510981 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.510918    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c444bca-6bc7-4d67-8e29-322ba9b044bd" containerName="maas-api"
Sep 07 22:58:06.510981 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.510927    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c444bca-6bc7-4d67-8e29-322ba9b044bd" containerName="maas-api"
Sep 07 22:58:06.510981 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.510938    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fc29fd-39ba-4396-a151-942c5da72480" containerName="payload-processing"
Sep 07 22:58:06.510981 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.510946    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fc29fd-39ba-4396-a151-942c5da72480" containerName="payload-processing"
Sep 07 22:58:06.510981 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.510955    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c3f9312-3ada-404e-97e2-addd7947d228" containerName="payload-pre-processing"
Sep 07 22:58:06.510981 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.510963    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c3f9312-3ada-404e-97e2-addd7947d228" containerName="payload-pre-processing"
Sep 07 22:58:06.511344 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.511058    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5efcdd91-a3f8-4b84-bcc6-afc8b82f8dde" containerName="payload-pre-processing"
Sep 07 22:58:06.511344 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.511070    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c444bca-6bc7-4d67-8e29-322ba9b044bd" containerName="maas-api"
Sep 07 22:58:06.511344 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.511080    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="918d99ef-bdb4-4618-9b40-9a601b24020d" containerName="payload-pre-processing"
Sep 07 22:58:06.511344 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.511089    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="11efd1f5-0a20-4c7a-841d-e87b11f492f7" containerName="payload-processing"
Sep 07 22:58:06.511344 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.511097    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="564bcfd9-41b6-4ee4-b553-44bccb29a8cd" containerName="maas-api"
Sep 07 22:58:06.511344 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.511105    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c3f9312-3ada-404e-97e2-addd7947d228" containerName="payload-pre-processing"
Sep 07 22:58:06.511344 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.511113    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="76aba709-84f3-4332-b14e-5c8f75f5ae66" containerName="payload-processing"
Sep 07 22:58:06.511344 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.511123    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fc29fd-39ba-4396-a151-942c5da72480" containerName="payload-processing"
Sep 07 22:58:06.516395 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.516364    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"
Sep 07 22:58:06.523859 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.523832    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-1dd653-serving-cert\""
Sep 07 22:58:06.531012 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.528017    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"]
Sep 07 22:58:06.643746 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.643708    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb98q\" (UniqueName: \"kubernetes.io/projected/27cb8308-22b9-49b9-bea5-95163b41a409-kube-api-access-fb98q\") pod \"maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl\" (UID: \"27cb8308-22b9-49b9-bea5-95163b41a409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"
Sep 07 22:58:06.643901 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.643753    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27cb8308-22b9-49b9-bea5-95163b41a409-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl\" (UID: \"27cb8308-22b9-49b9-bea5-95163b41a409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"
Sep 07 22:58:06.643901 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.643800    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27cb8308-22b9-49b9-bea5-95163b41a409-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl\" (UID: \"27cb8308-22b9-49b9-bea5-95163b41a409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"
Sep 07 22:58:06.643901 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.643888    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27cb8308-22b9-49b9-bea5-95163b41a409-metrics-certs\") pod \"maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl\" (UID: \"27cb8308-22b9-49b9-bea5-95163b41a409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"
Sep 07 22:58:06.744995 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.744954    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27cb8308-22b9-49b9-bea5-95163b41a409-metrics-certs\") pod \"maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl\" (UID: \"27cb8308-22b9-49b9-bea5-95163b41a409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"
Sep 07 22:58:06.745208 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.745010    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb98q\" (UniqueName: \"kubernetes.io/projected/27cb8308-22b9-49b9-bea5-95163b41a409-kube-api-access-fb98q\") pod \"maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl\" (UID: \"27cb8308-22b9-49b9-bea5-95163b41a409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"
Sep 07 22:58:06.745208 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.745046    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27cb8308-22b9-49b9-bea5-95163b41a409-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl\" (UID: \"27cb8308-22b9-49b9-bea5-95163b41a409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"
Sep 07 22:58:06.745208 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.745097    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27cb8308-22b9-49b9-bea5-95163b41a409-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl\" (UID: \"27cb8308-22b9-49b9-bea5-95163b41a409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"
Sep 07 22:58:06.745815 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.745787    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27cb8308-22b9-49b9-bea5-95163b41a409-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl\" (UID: \"27cb8308-22b9-49b9-bea5-95163b41a409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"
Sep 07 22:58:06.747685 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.747659    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27cb8308-22b9-49b9-bea5-95163b41a409-metrics-certs\") pod \"maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl\" (UID: \"27cb8308-22b9-49b9-bea5-95163b41a409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"
Sep 07 22:58:06.747789 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.747694    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27cb8308-22b9-49b9-bea5-95163b41a409-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl\" (UID: \"27cb8308-22b9-49b9-bea5-95163b41a409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"
Sep 07 22:58:06.753044 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.753015    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb98q\" (UniqueName: \"kubernetes.io/projected/27cb8308-22b9-49b9-bea5-95163b41a409-kube-api-access-fb98q\") pod \"maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl\" (UID: \"27cb8308-22b9-49b9-bea5-95163b41a409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"
Sep 07 22:58:06.838145 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:06.838107    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"
Sep 07 22:58:07.116201 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:07.116152    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"]
Sep 07 22:58:07.118611 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:58:07.118579    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27cb8308_22b9_49b9_bea5_95163b41a409.slice/crio-b0bf52ecc777a0a16000e99849af7338e102bec1d46afc203e4d428a1cd8d95b WatchSource:0}: Error finding container b0bf52ecc777a0a16000e99849af7338e102bec1d46afc203e4d428a1cd8d95b: Status 404 returned error can't find the container with id b0bf52ecc777a0a16000e99849af7338e102bec1d46afc203e4d428a1cd8d95b
Sep 07 22:58:07.170205 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:07.170161    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d"]
Sep 07 22:58:07.174988 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:07.174963    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d"
Sep 07 22:58:07.177389 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:07.177360    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1dd653\""
Sep 07 22:58:07.177506 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:07.177436    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1dd653-dockercfg-nf25j\""
Sep 07 22:58:07.184086 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:07.184054    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d"]
Sep 07 22:58:07.251134 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:07.251103    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xzc5\" (UniqueName: \"kubernetes.io/projected/68079a96-828c-4ccd-9b3f-a6c25f1d3297-kube-api-access-6xzc5\") pod \"payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d\" (UID: \"68079a96-828c-4ccd-9b3f-a6c25f1d3297\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d"
Sep 07 22:58:07.251318 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:07.251160    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68079a96-828c-4ccd-9b3f-a6c25f1d3297-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d\" (UID: \"68079a96-828c-4ccd-9b3f-a6c25f1d3297\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d"
Sep 07 22:58:07.351838 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:07.351796    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68079a96-828c-4ccd-9b3f-a6c25f1d3297-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d\" (UID: \"68079a96-828c-4ccd-9b3f-a6c25f1d3297\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d"
Sep 07 22:58:07.352039 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:07.351921    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xzc5\" (UniqueName: \"kubernetes.io/projected/68079a96-828c-4ccd-9b3f-a6c25f1d3297-kube-api-access-6xzc5\") pod \"payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d\" (UID: \"68079a96-828c-4ccd-9b3f-a6c25f1d3297\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d"
Sep 07 22:58:07.352649 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:07.352620    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68079a96-828c-4ccd-9b3f-a6c25f1d3297-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d\" (UID: \"68079a96-828c-4ccd-9b3f-a6c25f1d3297\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d"
Sep 07 22:58:07.360140 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:07.360109    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xzc5\" (UniqueName: \"kubernetes.io/projected/68079a96-828c-4ccd-9b3f-a6c25f1d3297-kube-api-access-6xzc5\") pod \"payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d\" (UID: \"68079a96-828c-4ccd-9b3f-a6c25f1d3297\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d"
Sep 07 22:58:07.487411 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:07.487313    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d"
Sep 07 22:58:07.582333 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:07.582281    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl" event={"ID":"27cb8308-22b9-49b9-bea5-95163b41a409","Type":"ContainerStarted","Data":"b0bf52ecc777a0a16000e99849af7338e102bec1d46afc203e4d428a1cd8d95b"}
Sep 07 22:58:07.761931 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:07.761905    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d"]
Sep 07 22:58:07.764728 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:58:07.764689    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68079a96_828c_4ccd_9b3f_a6c25f1d3297.slice/crio-4a53962f1a373ce4307ec8db6aa2e67bec2aede2b2a9c2bbc83cde6f4d03fa8a WatchSource:0}: Error finding container 4a53962f1a373ce4307ec8db6aa2e67bec2aede2b2a9c2bbc83cde6f4d03fa8a: Status 404 returned error can't find the container with id 4a53962f1a373ce4307ec8db6aa2e67bec2aede2b2a9c2bbc83cde6f4d03fa8a
Sep 07 22:58:08.591043 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:08.591004    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d" event={"ID":"68079a96-828c-4ccd-9b3f-a6c25f1d3297","Type":"ContainerStarted","Data":"48e52b9a07f30b90cb3ba17469f184bca3bd67b6fe9018665914029c0bd09626"}
Sep 07 22:58:08.591550 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:08.591070    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d" event={"ID":"68079a96-828c-4ccd-9b3f-a6c25f1d3297","Type":"ContainerStarted","Data":"4a53962f1a373ce4307ec8db6aa2e67bec2aede2b2a9c2bbc83cde6f4d03fa8a"}
Sep 07 22:58:08.591550 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:08.591198    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d"
Sep 07 22:58:08.593222 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:08.593195    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl" event={"ID":"27cb8308-22b9-49b9-bea5-95163b41a409","Type":"ContainerStarted","Data":"c44f9b5f2d05ac9b01c7237d7069bfdc42642bf4a93a52883a36176edd0ce10b"}
Sep 07 22:58:08.593395 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:08.593270    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"
Sep 07 22:58:08.609018 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:08.608966    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d" podStartSLOduration=1.608951493 podStartE2EDuration="1.608951493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:58:08.606402634 +0000 UTC m=+1571.663235598" watchObservedRunningTime="2026-09-07 22:58:08.608951493 +0000 UTC m=+1571.665784499"
Sep 07 22:58:08.624627 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:08.624565    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl" podStartSLOduration=2.193848531 podStartE2EDuration="2.624546116s" totalImagesPullingTime="430.697585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:58:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:58:08.622659329 +0000 UTC m=+1571.679492287" watchObservedRunningTime="2026-09-07 22:58:08.624546116 +0000 UTC m=+1571.681379076"
Sep 07 22:58:14.603901 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:14.603869    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"
Sep 07 22:58:19.601355 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:19.601318    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d"
Sep 07 22:58:35.868410 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:35.868372    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww"]
Sep 07 22:58:35.871680 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:35.871647    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww"
Sep 07 22:58:35.874190 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:35.874169    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-046f93-dockercfg-q58r6\""
Sep 07 22:58:35.874436 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:35.874422    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-046f93\""
Sep 07 22:58:35.882514 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:35.882484    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww"]
Sep 07 22:58:35.915235 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:35.915190    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc"]
Sep 07 22:58:35.918981 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:35.918955    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc"
Sep 07 22:58:35.921509 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:35.921481    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf9c8\" (UniqueName: \"kubernetes.io/projected/a8d71988-7a6a-4bb8-b2a0-0e9697aabb55-kube-api-access-xf9c8\") pod \"payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww\" (UID: \"a8d71988-7a6a-4bb8-b2a0-0e9697aabb55\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww"
Sep 07 22:58:35.921691 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:35.921516    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d71988-7a6a-4bb8-b2a0-0e9697aabb55-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww\" (UID: \"a8d71988-7a6a-4bb8-b2a0-0e9697aabb55\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww"
Sep 07 22:58:35.921691 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:35.921565    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fpfv\" (UniqueName: \"kubernetes.io/projected/ab8df4d3-0a6f-42cd-8875-331b60960df6-kube-api-access-5fpfv\") pod \"payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc\" (UID: \"ab8df4d3-0a6f-42cd-8875-331b60960df6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc"
Sep 07 22:58:35.921691 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:35.921661    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8df4d3-0a6f-42cd-8875-331b60960df6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc\" (UID: \"ab8df4d3-0a6f-42cd-8875-331b60960df6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc"
Sep 07 22:58:35.929315 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:35.929281    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc"]
Sep 07 22:58:36.022416 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.022374    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf9c8\" (UniqueName: \"kubernetes.io/projected/a8d71988-7a6a-4bb8-b2a0-0e9697aabb55-kube-api-access-xf9c8\") pod \"payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww\" (UID: \"a8d71988-7a6a-4bb8-b2a0-0e9697aabb55\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww"
Sep 07 22:58:36.022416 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.022421    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d71988-7a6a-4bb8-b2a0-0e9697aabb55-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww\" (UID: \"a8d71988-7a6a-4bb8-b2a0-0e9697aabb55\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww"
Sep 07 22:58:36.022691 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.022473    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fpfv\" (UniqueName: \"kubernetes.io/projected/ab8df4d3-0a6f-42cd-8875-331b60960df6-kube-api-access-5fpfv\") pod \"payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc\" (UID: \"ab8df4d3-0a6f-42cd-8875-331b60960df6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc"
Sep 07 22:58:36.022691 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.022547    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8df4d3-0a6f-42cd-8875-331b60960df6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc\" (UID: \"ab8df4d3-0a6f-42cd-8875-331b60960df6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc"
Sep 07 22:58:36.023295 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.023271    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8df4d3-0a6f-42cd-8875-331b60960df6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc\" (UID: \"ab8df4d3-0a6f-42cd-8875-331b60960df6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc"
Sep 07 22:58:36.023390 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.023304    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d71988-7a6a-4bb8-b2a0-0e9697aabb55-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww\" (UID: \"a8d71988-7a6a-4bb8-b2a0-0e9697aabb55\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww"
Sep 07 22:58:36.032084 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.032053    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf9c8\" (UniqueName: \"kubernetes.io/projected/a8d71988-7a6a-4bb8-b2a0-0e9697aabb55-kube-api-access-xf9c8\") pod \"payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww\" (UID: \"a8d71988-7a6a-4bb8-b2a0-0e9697aabb55\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww"
Sep 07 22:58:36.032216 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.032118    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fpfv\" (UniqueName: \"kubernetes.io/projected/ab8df4d3-0a6f-42cd-8875-331b60960df6-kube-api-access-5fpfv\") pod \"payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc\" (UID: \"ab8df4d3-0a6f-42cd-8875-331b60960df6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc"
Sep 07 22:58:36.184009 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.183914    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww"
Sep 07 22:58:36.231177 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.231133    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc"
Sep 07 22:58:36.460184 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.459937    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww"]
Sep 07 22:58:36.462644 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:58:36.462611    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d71988_7a6a_4bb8_b2a0_0e9697aabb55.slice/crio-74ce59d4ae741b6906eaa57840d98b81ea77537ef588f923d724a86340271999 WatchSource:0}: Error finding container 74ce59d4ae741b6906eaa57840d98b81ea77537ef588f923d724a86340271999: Status 404 returned error can't find the container with id 74ce59d4ae741b6906eaa57840d98b81ea77537ef588f923d724a86340271999
Sep 07 22:58:36.538307 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.538275    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc"]
Sep 07 22:58:36.540909 ip-10-0-138-99 kubenswrapper[2567]: W0907 22:58:36.540877    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab8df4d3_0a6f_42cd_8875_331b60960df6.slice/crio-63b5e91c0cf635238d5db70ad57261d19dae04d3d43464fce45599e62d867efc WatchSource:0}: Error finding container 63b5e91c0cf635238d5db70ad57261d19dae04d3d43464fce45599e62d867efc: Status 404 returned error can't find the container with id 63b5e91c0cf635238d5db70ad57261d19dae04d3d43464fce45599e62d867efc
Sep 07 22:58:36.734296 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.734171    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc" event={"ID":"ab8df4d3-0a6f-42cd-8875-331b60960df6","Type":"ContainerStarted","Data":"4ecf96da0abfa44d4c0891876913d841cf4a1a80569f8c6322aefb2a52227e24"}
Sep 07 22:58:36.734296 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.734235    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc" event={"ID":"ab8df4d3-0a6f-42cd-8875-331b60960df6","Type":"ContainerStarted","Data":"63b5e91c0cf635238d5db70ad57261d19dae04d3d43464fce45599e62d867efc"}
Sep 07 22:58:36.734542 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.734367    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc"
Sep 07 22:58:36.736042 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.736014    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww" event={"ID":"a8d71988-7a6a-4bb8-b2a0-0e9697aabb55","Type":"ContainerStarted","Data":"b6eabfb4ad0bfda817413c204e37b928e03de7fd781e30494baa30207cb0ecf3"}
Sep 07 22:58:36.736200 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.736069    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww"
Sep 07 22:58:36.736200 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.736091    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww" event={"ID":"a8d71988-7a6a-4bb8-b2a0-0e9697aabb55","Type":"ContainerStarted","Data":"74ce59d4ae741b6906eaa57840d98b81ea77537ef588f923d724a86340271999"}
Sep 07 22:58:36.751789 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.751723    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc" podStartSLOduration=1.751701207 podStartE2EDuration="1.751701207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:58:36.747971322 +0000 UTC m=+1599.804804300" watchObservedRunningTime="2026-09-07 22:58:36.751701207 +0000 UTC m=+1599.808534169"
Sep 07 22:58:36.763265 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:36.763183    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww" podStartSLOduration=1.763166231 podStartE2EDuration="1.763166231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:58:36.761738512 +0000 UTC m=+1599.818571471" watchObservedRunningTime="2026-09-07 22:58:36.763166231 +0000 UTC m=+1599.819999192"
Sep 07 22:58:47.744656 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:47.744623    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc"
Sep 07 22:58:47.745057 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:58:47.744696    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww"
Sep 07 22:59:04.139050 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:59:04.139010    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-6vqtn_d66e33da-96fe-433d-9dd2-3e5f1ef9f021/payload-processing/0.log"
Sep 07 22:59:04.258228 ip-10-0-138-99 kubenswrapper[2567]: I0907 22:59:04.258193    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc_ab8df4d3-0a6f-42cd-8875-331b60960df6/payload-processing/0.log"
Sep 07 23:00:00.175758 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:00.175721    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813700-mc5gl"]
Sep 07 23:00:00.179679 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:00.179647    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813700-mc5gl"
Sep 07 23:00:00.182530 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:00.182506    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 23:00:00.191200 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:00.191171    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813700-mc5gl"
Sep 07 23:00:00.207362 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:00.207277    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813700-mc5gl"]
Sep 07 23:00:00.469033 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:00.468872    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813700-mc5gl"]
Sep 07 23:00:00.471485 ip-10-0-138-99 kubenswrapper[2567]: W0907 23:00:00.471454    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719d805a_4232_4f00_b8de_c4fabae29c0d.slice/crio-fdcdbc4fed5dbff24d947a881e8aea49188f1e8d3d850d2321d53315f0d20d7f WatchSource:0}: Error finding container fdcdbc4fed5dbff24d947a881e8aea49188f1e8d3d850d2321d53315f0d20d7f: Status 404 returned error can't find the container with id fdcdbc4fed5dbff24d947a881e8aea49188f1e8d3d850d2321d53315f0d20d7f
Sep 07 23:00:01.171881 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:01.171841    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813700-mc5gl" event={"ID":"719d805a-4232-4f00-b8de-c4fabae29c0d","Type":"ContainerStarted","Data":"fdcdbc4fed5dbff24d947a881e8aea49188f1e8d3d850d2321d53315f0d20d7f"}
Sep 07 23:00:03.184799 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:03.184757    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813700-mc5gl" event={"ID":"719d805a-4232-4f00-b8de-c4fabae29c0d","Type":"ContainerStarted","Data":"e733be3d104e08a0568ec085bca66cfe612d9c36a7c55ab4ef197c066817be56"}
Sep 07 23:00:03.197478 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:03.197417    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813700-mc5gl" podStartSLOduration=1.009754027 podStartE2EDuration="3.197398863s" totalImagesPullingTime="2.187644836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 23:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:00:03.19680406 +0000 UTC m=+1686.253637020" watchObservedRunningTime="2026-09-07 23:00:03.197398863 +0000 UTC m=+1686.254231824"
Sep 07 23:00:04.192442 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:04.192404    2567 generic.go:356] "Generic (PLEG): container finished" podID="719d805a-4232-4f00-b8de-c4fabae29c0d" containerID="e733be3d104e08a0568ec085bca66cfe612d9c36a7c55ab4ef197c066817be56" exitCode=0
Sep 07 23:00:04.192828 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:04.192483    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813700-mc5gl" event={"ID":"719d805a-4232-4f00-b8de-c4fabae29c0d","Type":"ContainerDied","Data":"e733be3d104e08a0568ec085bca66cfe612d9c36a7c55ab4ef197c066817be56"}
Sep 07 23:00:04.564610 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:04.564579    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d_68079a96-828c-4ccd-9b3f-a6c25f1d3297/payload-processing/0.log"
Sep 07 23:00:04.685602 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:04.685559    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-6vqtn_d66e33da-96fe-433d-9dd2-3e5f1ef9f021/payload-processing/0.log"
Sep 07 23:00:04.805054 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:04.805006    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc_ab8df4d3-0a6f-42cd-8875-331b60960df6/payload-processing/0.log"
Sep 07 23:00:05.342419 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:05.342393    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813700-mc5gl"
Sep 07 23:00:06.204371 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:06.204335    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813700-mc5gl" event={"ID":"719d805a-4232-4f00-b8de-c4fabae29c0d","Type":"ContainerDied","Data":"fdcdbc4fed5dbff24d947a881e8aea49188f1e8d3d850d2321d53315f0d20d7f"}
Sep 07 23:00:06.204628 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:06.204375    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813700-mc5gl"
Sep 07 23:00:06.204628 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:06.204384    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdcdbc4fed5dbff24d947a881e8aea49188f1e8d3d850d2321d53315f0d20d7f"
Sep 07 23:00:09.911576 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:09.911535    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx"]
Sep 07 23:00:09.912015 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:09.912001    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="719d805a-4232-4f00-b8de-c4fabae29c0d" containerName="cleanup"
Sep 07 23:00:09.912066 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:09.912016    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="719d805a-4232-4f00-b8de-c4fabae29c0d" containerName="cleanup"
Sep 07 23:00:09.912102 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:09.912088    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="719d805a-4232-4f00-b8de-c4fabae29c0d" containerName="cleanup"
Sep 07 23:00:09.915468 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:09.915445    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx"
Sep 07 23:00:09.917644 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:09.917620    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-277d3e\""
Sep 07 23:00:09.917753 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:09.917734    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-277d3e-dockercfg-b6w76\""
Sep 07 23:00:09.923552 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:09.923528    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx"]
Sep 07 23:00:10.039558 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:10.039512    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8b327c-b8ad-4bec-b88b-4e66ab32462b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx\" (UID: \"5f8b327c-b8ad-4bec-b88b-4e66ab32462b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx"
Sep 07 23:00:10.039731 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:10.039691    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm8nw\" (UniqueName: \"kubernetes.io/projected/5f8b327c-b8ad-4bec-b88b-4e66ab32462b-kube-api-access-wm8nw\") pod \"payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx\" (UID: \"5f8b327c-b8ad-4bec-b88b-4e66ab32462b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx"
Sep 07 23:00:10.140912 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:10.140866    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm8nw\" (UniqueName: \"kubernetes.io/projected/5f8b327c-b8ad-4bec-b88b-4e66ab32462b-kube-api-access-wm8nw\") pod \"payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx\" (UID: \"5f8b327c-b8ad-4bec-b88b-4e66ab32462b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx"
Sep 07 23:00:10.140912 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:10.140924    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8b327c-b8ad-4bec-b88b-4e66ab32462b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx\" (UID: \"5f8b327c-b8ad-4bec-b88b-4e66ab32462b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx"
Sep 07 23:00:10.141657 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:10.141634    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8b327c-b8ad-4bec-b88b-4e66ab32462b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx\" (UID: \"5f8b327c-b8ad-4bec-b88b-4e66ab32462b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx"
Sep 07 23:00:10.148216 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:10.148176    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm8nw\" (UniqueName: \"kubernetes.io/projected/5f8b327c-b8ad-4bec-b88b-4e66ab32462b-kube-api-access-wm8nw\") pod \"payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx\" (UID: \"5f8b327c-b8ad-4bec-b88b-4e66ab32462b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx"
Sep 07 23:00:10.228620 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:10.228534    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx"
Sep 07 23:00:10.517891 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:10.517863    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx"]
Sep 07 23:00:10.519960 ip-10-0-138-99 kubenswrapper[2567]: W0907 23:00:10.519922    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f8b327c_b8ad_4bec_b88b_4e66ab32462b.slice/crio-3251bf21a12d777c0c621661652803a6fb5a92cc10f8a3d6493ead026fb7f076 WatchSource:0}: Error finding container 3251bf21a12d777c0c621661652803a6fb5a92cc10f8a3d6493ead026fb7f076: Status 404 returned error can't find the container with id 3251bf21a12d777c0c621661652803a6fb5a92cc10f8a3d6493ead026fb7f076
Sep 07 23:00:11.233203 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:11.233167    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx" event={"ID":"5f8b327c-b8ad-4bec-b88b-4e66ab32462b","Type":"ContainerStarted","Data":"379db7a8f49db8f8d899b3dc9cd81250545ddd32a3bee6dfbcbd4a31507d070c"}
Sep 07 23:00:11.233623 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:11.233221    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx" event={"ID":"5f8b327c-b8ad-4bec-b88b-4e66ab32462b","Type":"ContainerStarted","Data":"3251bf21a12d777c0c621661652803a6fb5a92cc10f8a3d6493ead026fb7f076"}
Sep 07 23:00:11.233623 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:11.233300    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx"
Sep 07 23:00:11.249859 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:11.249801    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx" podStartSLOduration=2.249782729 podStartE2EDuration="2.249782729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 23:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:00:11.248227954 +0000 UTC m=+1694.305060915" watchObservedRunningTime="2026-09-07 23:00:11.249782729 +0000 UTC m=+1694.306615720"
Sep 07 23:00:22.241223 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:22.241133    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx"
Sep 07 23:00:39.921877 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:39.921836    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx"]
Sep 07 23:00:39.922460 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:39.922090    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx" podUID="5f8b327c-b8ad-4bec-b88b-4e66ab32462b" containerName="payload-processing" containerID="cri-o://379db7a8f49db8f8d899b3dc9cd81250545ddd32a3bee6dfbcbd4a31507d070c" gracePeriod=30
Sep 07 23:00:40.188834 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:40.188767    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx"
Sep 07 23:00:40.237370 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:40.237331    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm8nw\" (UniqueName: \"kubernetes.io/projected/5f8b327c-b8ad-4bec-b88b-4e66ab32462b-kube-api-access-wm8nw\") pod \"5f8b327c-b8ad-4bec-b88b-4e66ab32462b\" (UID: \"5f8b327c-b8ad-4bec-b88b-4e66ab32462b\") "
Sep 07 23:00:40.237541 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:40.237497    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8b327c-b8ad-4bec-b88b-4e66ab32462b-plugins-config-volume\") pod \"5f8b327c-b8ad-4bec-b88b-4e66ab32462b\" (UID: \"5f8b327c-b8ad-4bec-b88b-4e66ab32462b\") "
Sep 07 23:00:40.237857 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:40.237834    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f8b327c-b8ad-4bec-b88b-4e66ab32462b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5f8b327c-b8ad-4bec-b88b-4e66ab32462b" (UID: "5f8b327c-b8ad-4bec-b88b-4e66ab32462b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:00:40.239472 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:40.239441    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f8b327c-b8ad-4bec-b88b-4e66ab32462b-kube-api-access-wm8nw" (OuterVolumeSpecName: "kube-api-access-wm8nw") pod "5f8b327c-b8ad-4bec-b88b-4e66ab32462b" (UID: "5f8b327c-b8ad-4bec-b88b-4e66ab32462b"). InnerVolumeSpecName "kube-api-access-wm8nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:00:40.338346 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:40.338310    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8b327c-b8ad-4bec-b88b-4e66ab32462b-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:00:40.338346 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:40.338341    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm8nw\" (UniqueName: \"kubernetes.io/projected/5f8b327c-b8ad-4bec-b88b-4e66ab32462b-kube-api-access-wm8nw\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:00:40.384550 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:40.384508    2567 generic.go:356] "Generic (PLEG): container finished" podID="5f8b327c-b8ad-4bec-b88b-4e66ab32462b" containerID="379db7a8f49db8f8d899b3dc9cd81250545ddd32a3bee6dfbcbd4a31507d070c" exitCode=0
Sep 07 23:00:40.384718 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:40.384568    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx" event={"ID":"5f8b327c-b8ad-4bec-b88b-4e66ab32462b","Type":"ContainerDied","Data":"379db7a8f49db8f8d899b3dc9cd81250545ddd32a3bee6dfbcbd4a31507d070c"}
Sep 07 23:00:40.384718 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:40.384609    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx" event={"ID":"5f8b327c-b8ad-4bec-b88b-4e66ab32462b","Type":"ContainerDied","Data":"3251bf21a12d777c0c621661652803a6fb5a92cc10f8a3d6493ead026fb7f076"}
Sep 07 23:00:40.384718 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:40.384631    2567 scope.go:117] "RemoveContainer" containerID="379db7a8f49db8f8d899b3dc9cd81250545ddd32a3bee6dfbcbd4a31507d070c"
Sep 07 23:00:40.384718 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:40.384567    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx"
Sep 07 23:00:40.394461 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:40.394437    2567 scope.go:117] "RemoveContainer" containerID="379db7a8f49db8f8d899b3dc9cd81250545ddd32a3bee6dfbcbd4a31507d070c"
Sep 07 23:00:40.394809 ip-10-0-138-99 kubenswrapper[2567]: E0907 23:00:40.394788    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379db7a8f49db8f8d899b3dc9cd81250545ddd32a3bee6dfbcbd4a31507d070c\": container with ID starting with 379db7a8f49db8f8d899b3dc9cd81250545ddd32a3bee6dfbcbd4a31507d070c not found: ID does not exist" containerID="379db7a8f49db8f8d899b3dc9cd81250545ddd32a3bee6dfbcbd4a31507d070c"
Sep 07 23:00:40.394860 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:40.394828    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379db7a8f49db8f8d899b3dc9cd81250545ddd32a3bee6dfbcbd4a31507d070c"} err="failed to get container status \"379db7a8f49db8f8d899b3dc9cd81250545ddd32a3bee6dfbcbd4a31507d070c\": rpc error: code = NotFound desc = could not find container \"379db7a8f49db8f8d899b3dc9cd81250545ddd32a3bee6dfbcbd4a31507d070c\": container with ID starting with 379db7a8f49db8f8d899b3dc9cd81250545ddd32a3bee6dfbcbd4a31507d070c not found: ID does not exist"
Sep 07 23:00:40.405451 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:40.405416    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx"]
Sep 07 23:00:40.409364 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:40.409335    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-277d3e-5d8745c6f5-djcqx"]
Sep 07 23:00:41.466337 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:00:41.466299    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f8b327c-b8ad-4bec-b88b-4e66ab32462b" path="/var/lib/kubelet/pods/5f8b327c-b8ad-4bec-b88b-4e66ab32462b/volumes"
Sep 07 23:01:05.360285 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.360220    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"]
Sep 07 23:01:05.360819 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.360498    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl" podUID="27cb8308-22b9-49b9-bea5-95163b41a409" containerName="maas-api" containerID="cri-o://c44f9b5f2d05ac9b01c7237d7069bfdc42642bf4a93a52883a36176edd0ce10b" gracePeriod=30
Sep 07 23:01:05.418210 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.418163    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d"]
Sep 07 23:01:05.418517 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.418455    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d" podUID="68079a96-828c-4ccd-9b3f-a6c25f1d3297" containerName="payload-processing" containerID="cri-o://48e52b9a07f30b90cb3ba17469f184bca3bd67b6fe9018665914029c0bd09626" gracePeriod=30
Sep 07 23:01:05.517946 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.517904    2567 generic.go:356] "Generic (PLEG): container finished" podID="27cb8308-22b9-49b9-bea5-95163b41a409" containerID="c44f9b5f2d05ac9b01c7237d7069bfdc42642bf4a93a52883a36176edd0ce10b" exitCode=0
Sep 07 23:01:05.518103 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.517976    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl" event={"ID":"27cb8308-22b9-49b9-bea5-95163b41a409","Type":"ContainerDied","Data":"c44f9b5f2d05ac9b01c7237d7069bfdc42642bf4a93a52883a36176edd0ce10b"}
Sep 07 23:01:05.683568 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.683542    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"
Sep 07 23:01:05.715503 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.715475    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d"
Sep 07 23:01:05.795371 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.795328    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27cb8308-22b9-49b9-bea5-95163b41a409-metrics-certs\") pod \"27cb8308-22b9-49b9-bea5-95163b41a409\" (UID: \"27cb8308-22b9-49b9-bea5-95163b41a409\") "
Sep 07 23:01:05.795371 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.795369    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xzc5\" (UniqueName: \"kubernetes.io/projected/68079a96-828c-4ccd-9b3f-a6c25f1d3297-kube-api-access-6xzc5\") pod \"68079a96-828c-4ccd-9b3f-a6c25f1d3297\" (UID: \"68079a96-828c-4ccd-9b3f-a6c25f1d3297\") "
Sep 07 23:01:05.795636 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.795393    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27cb8308-22b9-49b9-bea5-95163b41a409-maas-api-tls\") pod \"27cb8308-22b9-49b9-bea5-95163b41a409\" (UID: \"27cb8308-22b9-49b9-bea5-95163b41a409\") "
Sep 07 23:01:05.795636 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.795425    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27cb8308-22b9-49b9-bea5-95163b41a409-openshift-service-ca\") pod \"27cb8308-22b9-49b9-bea5-95163b41a409\" (UID: \"27cb8308-22b9-49b9-bea5-95163b41a409\") "
Sep 07 23:01:05.795636 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.795572    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fb98q\" (UniqueName: \"kubernetes.io/projected/27cb8308-22b9-49b9-bea5-95163b41a409-kube-api-access-fb98q\") pod \"27cb8308-22b9-49b9-bea5-95163b41a409\" (UID: \"27cb8308-22b9-49b9-bea5-95163b41a409\") "
Sep 07 23:01:05.795636 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.795624    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68079a96-828c-4ccd-9b3f-a6c25f1d3297-plugins-config-volume\") pod \"68079a96-828c-4ccd-9b3f-a6c25f1d3297\" (UID: \"68079a96-828c-4ccd-9b3f-a6c25f1d3297\") "
Sep 07 23:01:05.795864 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.795771    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27cb8308-22b9-49b9-bea5-95163b41a409-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "27cb8308-22b9-49b9-bea5-95163b41a409" (UID: "27cb8308-22b9-49b9-bea5-95163b41a409"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:01:05.796184 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.796082    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27cb8308-22b9-49b9-bea5-95163b41a409-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:01:05.796184 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.796122    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68079a96-828c-4ccd-9b3f-a6c25f1d3297-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "68079a96-828c-4ccd-9b3f-a6c25f1d3297" (UID: "68079a96-828c-4ccd-9b3f-a6c25f1d3297"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:01:05.797695 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.797664    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27cb8308-22b9-49b9-bea5-95163b41a409-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "27cb8308-22b9-49b9-bea5-95163b41a409" (UID: "27cb8308-22b9-49b9-bea5-95163b41a409"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:01:05.797801 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.797690    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27cb8308-22b9-49b9-bea5-95163b41a409-kube-api-access-fb98q" (OuterVolumeSpecName: "kube-api-access-fb98q") pod "27cb8308-22b9-49b9-bea5-95163b41a409" (UID: "27cb8308-22b9-49b9-bea5-95163b41a409"). InnerVolumeSpecName "kube-api-access-fb98q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:01:05.797801 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.797732    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27cb8308-22b9-49b9-bea5-95163b41a409-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "27cb8308-22b9-49b9-bea5-95163b41a409" (UID: "27cb8308-22b9-49b9-bea5-95163b41a409"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:01:05.797801 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.797750    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68079a96-828c-4ccd-9b3f-a6c25f1d3297-kube-api-access-6xzc5" (OuterVolumeSpecName: "kube-api-access-6xzc5") pod "68079a96-828c-4ccd-9b3f-a6c25f1d3297" (UID: "68079a96-828c-4ccd-9b3f-a6c25f1d3297"). InnerVolumeSpecName "kube-api-access-6xzc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:01:05.898670 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.898582    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27cb8308-22b9-49b9-bea5-95163b41a409-metrics-certs\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:01:05.898670 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.898613    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xzc5\" (UniqueName: \"kubernetes.io/projected/68079a96-828c-4ccd-9b3f-a6c25f1d3297-kube-api-access-6xzc5\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:01:05.898670 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.898625    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27cb8308-22b9-49b9-bea5-95163b41a409-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:01:05.898670 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.898634    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb98q\" (UniqueName: \"kubernetes.io/projected/27cb8308-22b9-49b9-bea5-95163b41a409-kube-api-access-fb98q\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:01:05.898670 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:05.898643    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68079a96-828c-4ccd-9b3f-a6c25f1d3297-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:01:06.524213 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:06.524179    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"
Sep 07 23:01:06.524815 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:06.524214    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl" event={"ID":"27cb8308-22b9-49b9-bea5-95163b41a409","Type":"ContainerDied","Data":"b0bf52ecc777a0a16000e99849af7338e102bec1d46afc203e4d428a1cd8d95b"}
Sep 07 23:01:06.524815 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:06.524307    2567 scope.go:117] "RemoveContainer" containerID="c44f9b5f2d05ac9b01c7237d7069bfdc42642bf4a93a52883a36176edd0ce10b"
Sep 07 23:01:06.525743 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:06.525597    2567 generic.go:356] "Generic (PLEG): container finished" podID="68079a96-828c-4ccd-9b3f-a6c25f1d3297" containerID="48e52b9a07f30b90cb3ba17469f184bca3bd67b6fe9018665914029c0bd09626" exitCode=0
Sep 07 23:01:06.525743 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:06.525627    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d" event={"ID":"68079a96-828c-4ccd-9b3f-a6c25f1d3297","Type":"ContainerDied","Data":"48e52b9a07f30b90cb3ba17469f184bca3bd67b6fe9018665914029c0bd09626"}
Sep 07 23:01:06.525743 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:06.525648    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d"
Sep 07 23:01:06.525743 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:06.525664    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d" event={"ID":"68079a96-828c-4ccd-9b3f-a6c25f1d3297","Type":"ContainerDied","Data":"4a53962f1a373ce4307ec8db6aa2e67bec2aede2b2a9c2bbc83cde6f4d03fa8a"}
Sep 07 23:01:06.534523 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:06.534497    2567 scope.go:117] "RemoveContainer" containerID="48e52b9a07f30b90cb3ba17469f184bca3bd67b6fe9018665914029c0bd09626"
Sep 07 23:01:06.543163 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:06.543138    2567 scope.go:117] "RemoveContainer" containerID="48e52b9a07f30b90cb3ba17469f184bca3bd67b6fe9018665914029c0bd09626"
Sep 07 23:01:06.543513 ip-10-0-138-99 kubenswrapper[2567]: E0907 23:01:06.543491    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48e52b9a07f30b90cb3ba17469f184bca3bd67b6fe9018665914029c0bd09626\": container with ID starting with 48e52b9a07f30b90cb3ba17469f184bca3bd67b6fe9018665914029c0bd09626 not found: ID does not exist" containerID="48e52b9a07f30b90cb3ba17469f184bca3bd67b6fe9018665914029c0bd09626"
Sep 07 23:01:06.543600 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:06.543520    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e52b9a07f30b90cb3ba17469f184bca3bd67b6fe9018665914029c0bd09626"} err="failed to get container status \"48e52b9a07f30b90cb3ba17469f184bca3bd67b6fe9018665914029c0bd09626\": rpc error: code = NotFound desc = could not find container \"48e52b9a07f30b90cb3ba17469f184bca3bd67b6fe9018665914029c0bd09626\": container with ID starting with 48e52b9a07f30b90cb3ba17469f184bca3bd67b6fe9018665914029c0bd09626 not found: ID does not exist"
Sep 07 23:01:06.547537 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:06.547511    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d"]
Sep 07 23:01:06.558016 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:06.557981    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1dd653-5f6785696c-95m2d"]
Sep 07 23:01:06.570517 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:06.570484    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"]
Sep 07 23:01:06.575949 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:06.575919    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dd653-84b5469864-vkjpl"]
Sep 07 23:01:07.466319 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:07.466281    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27cb8308-22b9-49b9-bea5-95163b41a409" path="/var/lib/kubelet/pods/27cb8308-22b9-49b9-bea5-95163b41a409/volumes"
Sep 07 23:01:07.466677 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:07.466663    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68079a96-828c-4ccd-9b3f-a6c25f1d3297" path="/var/lib/kubelet/pods/68079a96-828c-4ccd-9b3f-a6c25f1d3297/volumes"
Sep 07 23:01:57.541185 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:57.541155    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/2.log"
Sep 07 23:01:57.545529 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:01:57.545501    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/2.log"
Sep 07 23:02:09.874365 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:09.874326    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc"]
Sep 07 23:02:09.874893 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:09.874617    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc" podUID="ab8df4d3-0a6f-42cd-8875-331b60960df6" containerName="payload-processing" containerID="cri-o://4ecf96da0abfa44d4c0891876913d841cf4a1a80569f8c6322aefb2a52227e24" gracePeriod=30
Sep 07 23:02:09.881594 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:09.881562    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww"]
Sep 07 23:02:09.881892 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:09.881798    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww" podUID="a8d71988-7a6a-4bb8-b2a0-0e9697aabb55" containerName="payload-pre-processing" containerID="cri-o://b6eabfb4ad0bfda817413c204e37b928e03de7fd781e30494baa30207cb0ecf3" gracePeriod=30
Sep 07 23:02:10.138856 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.138786    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc"
Sep 07 23:02:10.170238 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.170212    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww"
Sep 07 23:02:10.222325 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.222282    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fpfv\" (UniqueName: \"kubernetes.io/projected/ab8df4d3-0a6f-42cd-8875-331b60960df6-kube-api-access-5fpfv\") pod \"ab8df4d3-0a6f-42cd-8875-331b60960df6\" (UID: \"ab8df4d3-0a6f-42cd-8875-331b60960df6\") "
Sep 07 23:02:10.222503 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.222371    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8df4d3-0a6f-42cd-8875-331b60960df6-plugins-config-volume\") pod \"ab8df4d3-0a6f-42cd-8875-331b60960df6\" (UID: \"ab8df4d3-0a6f-42cd-8875-331b60960df6\") "
Sep 07 23:02:10.222743 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.222712    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab8df4d3-0a6f-42cd-8875-331b60960df6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ab8df4d3-0a6f-42cd-8875-331b60960df6" (UID: "ab8df4d3-0a6f-42cd-8875-331b60960df6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:02:10.224499 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.224464    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8df4d3-0a6f-42cd-8875-331b60960df6-kube-api-access-5fpfv" (OuterVolumeSpecName: "kube-api-access-5fpfv") pod "ab8df4d3-0a6f-42cd-8875-331b60960df6" (UID: "ab8df4d3-0a6f-42cd-8875-331b60960df6"). InnerVolumeSpecName "kube-api-access-5fpfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:02:10.323946 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.323906    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d71988-7a6a-4bb8-b2a0-0e9697aabb55-plugins-config-volume\") pod \"a8d71988-7a6a-4bb8-b2a0-0e9697aabb55\" (UID: \"a8d71988-7a6a-4bb8-b2a0-0e9697aabb55\") "
Sep 07 23:02:10.323946 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.323955    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf9c8\" (UniqueName: \"kubernetes.io/projected/a8d71988-7a6a-4bb8-b2a0-0e9697aabb55-kube-api-access-xf9c8\") pod \"a8d71988-7a6a-4bb8-b2a0-0e9697aabb55\" (UID: \"a8d71988-7a6a-4bb8-b2a0-0e9697aabb55\") "
Sep 07 23:02:10.324211 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.324133    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fpfv\" (UniqueName: \"kubernetes.io/projected/ab8df4d3-0a6f-42cd-8875-331b60960df6-kube-api-access-5fpfv\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:02:10.324211 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.324143    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8df4d3-0a6f-42cd-8875-331b60960df6-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:02:10.324389 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.324327    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d71988-7a6a-4bb8-b2a0-0e9697aabb55-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8d71988-7a6a-4bb8-b2a0-0e9697aabb55" (UID: "a8d71988-7a6a-4bb8-b2a0-0e9697aabb55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:02:10.326129 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.326102    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d71988-7a6a-4bb8-b2a0-0e9697aabb55-kube-api-access-xf9c8" (OuterVolumeSpecName: "kube-api-access-xf9c8") pod "a8d71988-7a6a-4bb8-b2a0-0e9697aabb55" (UID: "a8d71988-7a6a-4bb8-b2a0-0e9697aabb55"). InnerVolumeSpecName "kube-api-access-xf9c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:02:10.425652 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.425551    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf9c8\" (UniqueName: \"kubernetes.io/projected/a8d71988-7a6a-4bb8-b2a0-0e9697aabb55-kube-api-access-xf9c8\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:02:10.425652 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.425589    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d71988-7a6a-4bb8-b2a0-0e9697aabb55-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:02:10.857674 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.857637    2567 generic.go:356] "Generic (PLEG): container finished" podID="ab8df4d3-0a6f-42cd-8875-331b60960df6" containerID="4ecf96da0abfa44d4c0891876913d841cf4a1a80569f8c6322aefb2a52227e24" exitCode=0
Sep 07 23:02:10.857880 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.857687    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc"
Sep 07 23:02:10.857880 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.857720    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc" event={"ID":"ab8df4d3-0a6f-42cd-8875-331b60960df6","Type":"ContainerDied","Data":"4ecf96da0abfa44d4c0891876913d841cf4a1a80569f8c6322aefb2a52227e24"}
Sep 07 23:02:10.857880 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.857768    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc" event={"ID":"ab8df4d3-0a6f-42cd-8875-331b60960df6","Type":"ContainerDied","Data":"63b5e91c0cf635238d5db70ad57261d19dae04d3d43464fce45599e62d867efc"}
Sep 07 23:02:10.857880 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.857789    2567 scope.go:117] "RemoveContainer" containerID="4ecf96da0abfa44d4c0891876913d841cf4a1a80569f8c6322aefb2a52227e24"
Sep 07 23:02:10.859173 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.859144    2567 generic.go:356] "Generic (PLEG): container finished" podID="a8d71988-7a6a-4bb8-b2a0-0e9697aabb55" containerID="b6eabfb4ad0bfda817413c204e37b928e03de7fd781e30494baa30207cb0ecf3" exitCode=0
Sep 07 23:02:10.859276 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.859237    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww"
Sep 07 23:02:10.859276 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.859232    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww" event={"ID":"a8d71988-7a6a-4bb8-b2a0-0e9697aabb55","Type":"ContainerDied","Data":"b6eabfb4ad0bfda817413c204e37b928e03de7fd781e30494baa30207cb0ecf3"}
Sep 07 23:02:10.859367 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.859305    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww" event={"ID":"a8d71988-7a6a-4bb8-b2a0-0e9697aabb55","Type":"ContainerDied","Data":"74ce59d4ae741b6906eaa57840d98b81ea77537ef588f923d724a86340271999"}
Sep 07 23:02:10.867719 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.867702    2567 scope.go:117] "RemoveContainer" containerID="4ecf96da0abfa44d4c0891876913d841cf4a1a80569f8c6322aefb2a52227e24"
Sep 07 23:02:10.868044 ip-10-0-138-99 kubenswrapper[2567]: E0907 23:02:10.868028    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ecf96da0abfa44d4c0891876913d841cf4a1a80569f8c6322aefb2a52227e24\": container with ID starting with 4ecf96da0abfa44d4c0891876913d841cf4a1a80569f8c6322aefb2a52227e24 not found: ID does not exist" containerID="4ecf96da0abfa44d4c0891876913d841cf4a1a80569f8c6322aefb2a52227e24"
Sep 07 23:02:10.868086 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.868052    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ecf96da0abfa44d4c0891876913d841cf4a1a80569f8c6322aefb2a52227e24"} err="failed to get container status \"4ecf96da0abfa44d4c0891876913d841cf4a1a80569f8c6322aefb2a52227e24\": rpc error: code = NotFound desc = could not find container \"4ecf96da0abfa44d4c0891876913d841cf4a1a80569f8c6322aefb2a52227e24\": container with ID starting with 4ecf96da0abfa44d4c0891876913d841cf4a1a80569f8c6322aefb2a52227e24 not found: ID does not exist"
Sep 07 23:02:10.868086 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.868070    2567 scope.go:117] "RemoveContainer" containerID="b6eabfb4ad0bfda817413c204e37b928e03de7fd781e30494baa30207cb0ecf3"
Sep 07 23:02:10.877550 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.877162    2567 scope.go:117] "RemoveContainer" containerID="b6eabfb4ad0bfda817413c204e37b928e03de7fd781e30494baa30207cb0ecf3"
Sep 07 23:02:10.877769 ip-10-0-138-99 kubenswrapper[2567]: E0907 23:02:10.877548    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6eabfb4ad0bfda817413c204e37b928e03de7fd781e30494baa30207cb0ecf3\": container with ID starting with b6eabfb4ad0bfda817413c204e37b928e03de7fd781e30494baa30207cb0ecf3 not found: ID does not exist" containerID="b6eabfb4ad0bfda817413c204e37b928e03de7fd781e30494baa30207cb0ecf3"
Sep 07 23:02:10.877769 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.877582    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6eabfb4ad0bfda817413c204e37b928e03de7fd781e30494baa30207cb0ecf3"} err="failed to get container status \"b6eabfb4ad0bfda817413c204e37b928e03de7fd781e30494baa30207cb0ecf3\": rpc error: code = NotFound desc = could not find container \"b6eabfb4ad0bfda817413c204e37b928e03de7fd781e30494baa30207cb0ecf3\": container with ID starting with b6eabfb4ad0bfda817413c204e37b928e03de7fd781e30494baa30207cb0ecf3 not found: ID does not exist"
Sep 07 23:02:10.879118 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.879095    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc"]
Sep 07 23:02:10.884986 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.884955    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-046f93-596767644c-wg5nc"]
Sep 07 23:02:10.894326 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.894298    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww"]
Sep 07 23:02:10.896079 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:10.896057    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-046f93-d5f5985fd-52fww"]
Sep 07 23:02:11.467222 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:11.467186    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8d71988-7a6a-4bb8-b2a0-0e9697aabb55" path="/var/lib/kubelet/pods/a8d71988-7a6a-4bb8-b2a0-0e9697aabb55/volumes"
Sep 07 23:02:11.467632 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:11.467618    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab8df4d3-0a6f-42cd-8875-331b60960df6" path="/var/lib/kubelet/pods/ab8df4d3-0a6f-42cd-8875-331b60960df6/volumes"
Sep 07 23:02:26.960285 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:26.959768    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx"]
Sep 07 23:02:26.960285 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:26.960041    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx" podUID="2c6c3708-549a-4e52-87fa-8ae773a62bdd" containerName="payload-pre-processing" containerID="cri-o://136c5c52cbeb9240b0b08d475d232b52bc9c5e5778ed1aefa1ffa355b8faafc1" gracePeriod=30
Sep 07 23:02:27.223770 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:27.223698    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx"
Sep 07 23:02:27.283819 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:27.283782    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75928\" (UniqueName: \"kubernetes.io/projected/2c6c3708-549a-4e52-87fa-8ae773a62bdd-kube-api-access-75928\") pod \"2c6c3708-549a-4e52-87fa-8ae773a62bdd\" (UID: \"2c6c3708-549a-4e52-87fa-8ae773a62bdd\") "
Sep 07 23:02:27.284002 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:27.283886    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6c3708-549a-4e52-87fa-8ae773a62bdd-plugins-config-volume\") pod \"2c6c3708-549a-4e52-87fa-8ae773a62bdd\" (UID: \"2c6c3708-549a-4e52-87fa-8ae773a62bdd\") "
Sep 07 23:02:27.284311 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:27.284285    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c6c3708-549a-4e52-87fa-8ae773a62bdd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c6c3708-549a-4e52-87fa-8ae773a62bdd" (UID: "2c6c3708-549a-4e52-87fa-8ae773a62bdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:02:27.286162 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:27.286135    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c6c3708-549a-4e52-87fa-8ae773a62bdd-kube-api-access-75928" (OuterVolumeSpecName: "kube-api-access-75928") pod "2c6c3708-549a-4e52-87fa-8ae773a62bdd" (UID: "2c6c3708-549a-4e52-87fa-8ae773a62bdd"). InnerVolumeSpecName "kube-api-access-75928". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:02:27.385020 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:27.384976    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6c3708-549a-4e52-87fa-8ae773a62bdd-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:02:27.385020 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:27.385015    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75928\" (UniqueName: \"kubernetes.io/projected/2c6c3708-549a-4e52-87fa-8ae773a62bdd-kube-api-access-75928\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:02:27.951938 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:27.951904    2567 generic.go:356] "Generic (PLEG): container finished" podID="2c6c3708-549a-4e52-87fa-8ae773a62bdd" containerID="136c5c52cbeb9240b0b08d475d232b52bc9c5e5778ed1aefa1ffa355b8faafc1" exitCode=0
Sep 07 23:02:27.952127 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:27.951951    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx"
Sep 07 23:02:27.952127 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:27.951992    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx" event={"ID":"2c6c3708-549a-4e52-87fa-8ae773a62bdd","Type":"ContainerDied","Data":"136c5c52cbeb9240b0b08d475d232b52bc9c5e5778ed1aefa1ffa355b8faafc1"}
Sep 07 23:02:27.952127 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:27.952043    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx" event={"ID":"2c6c3708-549a-4e52-87fa-8ae773a62bdd","Type":"ContainerDied","Data":"9ca6b1ba49bcd9cec399048be9b2ea71d093bdac3df1eee0a1108926e59acea9"}
Sep 07 23:02:27.952127 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:27.952067    2567 scope.go:117] "RemoveContainer" containerID="136c5c52cbeb9240b0b08d475d232b52bc9c5e5778ed1aefa1ffa355b8faafc1"
Sep 07 23:02:27.961236 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:27.961025    2567 scope.go:117] "RemoveContainer" containerID="136c5c52cbeb9240b0b08d475d232b52bc9c5e5778ed1aefa1ffa355b8faafc1"
Sep 07 23:02:27.963034 ip-10-0-138-99 kubenswrapper[2567]: E0907 23:02:27.963010    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"136c5c52cbeb9240b0b08d475d232b52bc9c5e5778ed1aefa1ffa355b8faafc1\": container with ID starting with 136c5c52cbeb9240b0b08d475d232b52bc9c5e5778ed1aefa1ffa355b8faafc1 not found: ID does not exist" containerID="136c5c52cbeb9240b0b08d475d232b52bc9c5e5778ed1aefa1ffa355b8faafc1"
Sep 07 23:02:27.963119 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:27.963040    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136c5c52cbeb9240b0b08d475d232b52bc9c5e5778ed1aefa1ffa355b8faafc1"} err="failed to get container status \"136c5c52cbeb9240b0b08d475d232b52bc9c5e5778ed1aefa1ffa355b8faafc1\": rpc error: code = NotFound desc = could not find container \"136c5c52cbeb9240b0b08d475d232b52bc9c5e5778ed1aefa1ffa355b8faafc1\": container with ID starting with 136c5c52cbeb9240b0b08d475d232b52bc9c5e5778ed1aefa1ffa355b8faafc1 not found: ID does not exist"
Sep 07 23:02:27.966975 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:27.966952    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx"]
Sep 07 23:02:27.970435 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:27.970407    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d0395-6f8cb64446-nggzx"]
Sep 07 23:02:29.467407 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:29.467306    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c6c3708-549a-4e52-87fa-8ae773a62bdd" path="/var/lib/kubelet/pods/2c6c3708-549a-4e52-87fa-8ae773a62bdd/volumes"
Sep 07 23:02:44.338676 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:44.338627    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr"]
Sep 07 23:02:44.339237 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:44.338888    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr" podUID="6ab5cdb6-fa92-4787-ba43-2dbaaebfb331" containerName="payload-processing" containerID="cri-o://2ff0e7f27b12fa178eacd801ce10786cf6af137424e431abe32ced2f53d3eff5" gracePeriod=30
Sep 07 23:02:44.346616 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:44.346583    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl"]
Sep 07 23:02:44.346799 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:44.346763    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl" podUID="c0f9b192-ba1b-431c-80e6-d7f35c95ecb8" containerName="payload-pre-processing" containerID="cri-o://8c70550af69534debe632133d61b9340d4b7a851a92b51e8166839c56c9de5a1" gracePeriod=30
Sep 07 23:02:44.621589 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:44.621564    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr"
Sep 07 23:02:44.624900 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:44.624872    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl"
Sep 07 23:02:44.750267 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:44.750214    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f9b192-ba1b-431c-80e6-d7f35c95ecb8-plugins-config-volume\") pod \"c0f9b192-ba1b-431c-80e6-d7f35c95ecb8\" (UID: \"c0f9b192-ba1b-431c-80e6-d7f35c95ecb8\") "
Sep 07 23:02:44.750440 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:44.750321    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5cdb6-fa92-4787-ba43-2dbaaebfb331-plugins-config-volume\") pod \"6ab5cdb6-fa92-4787-ba43-2dbaaebfb331\" (UID: \"6ab5cdb6-fa92-4787-ba43-2dbaaebfb331\") "
Sep 07 23:02:44.750440 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:44.750355    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnsfg\" (UniqueName: \"kubernetes.io/projected/6ab5cdb6-fa92-4787-ba43-2dbaaebfb331-kube-api-access-jnsfg\") pod \"6ab5cdb6-fa92-4787-ba43-2dbaaebfb331\" (UID: \"6ab5cdb6-fa92-4787-ba43-2dbaaebfb331\") "
Sep 07 23:02:44.750440 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:44.750373    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjlct\" (UniqueName: \"kubernetes.io/projected/c0f9b192-ba1b-431c-80e6-d7f35c95ecb8-kube-api-access-bjlct\") pod \"c0f9b192-ba1b-431c-80e6-d7f35c95ecb8\" (UID: \"c0f9b192-ba1b-431c-80e6-d7f35c95ecb8\") "
Sep 07 23:02:44.750722 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:44.750689    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0f9b192-ba1b-431c-80e6-d7f35c95ecb8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c0f9b192-ba1b-431c-80e6-d7f35c95ecb8" (UID: "c0f9b192-ba1b-431c-80e6-d7f35c95ecb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:02:44.750722 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:44.750710    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab5cdb6-fa92-4787-ba43-2dbaaebfb331-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ab5cdb6-fa92-4787-ba43-2dbaaebfb331" (UID: "6ab5cdb6-fa92-4787-ba43-2dbaaebfb331"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:02:44.752631 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:44.752600    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f9b192-ba1b-431c-80e6-d7f35c95ecb8-kube-api-access-bjlct" (OuterVolumeSpecName: "kube-api-access-bjlct") pod "c0f9b192-ba1b-431c-80e6-d7f35c95ecb8" (UID: "c0f9b192-ba1b-431c-80e6-d7f35c95ecb8"). InnerVolumeSpecName "kube-api-access-bjlct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:02:44.752739 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:44.752677    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab5cdb6-fa92-4787-ba43-2dbaaebfb331-kube-api-access-jnsfg" (OuterVolumeSpecName: "kube-api-access-jnsfg") pod "6ab5cdb6-fa92-4787-ba43-2dbaaebfb331" (UID: "6ab5cdb6-fa92-4787-ba43-2dbaaebfb331"). InnerVolumeSpecName "kube-api-access-jnsfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:02:44.851454 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:44.851370    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5cdb6-fa92-4787-ba43-2dbaaebfb331-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:02:44.851454 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:44.851403    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnsfg\" (UniqueName: \"kubernetes.io/projected/6ab5cdb6-fa92-4787-ba43-2dbaaebfb331-kube-api-access-jnsfg\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:02:44.851454 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:44.851412    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjlct\" (UniqueName: \"kubernetes.io/projected/c0f9b192-ba1b-431c-80e6-d7f35c95ecb8-kube-api-access-bjlct\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:02:44.851454 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:44.851422    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f9b192-ba1b-431c-80e6-d7f35c95ecb8-plugins-config-volume\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:02:45.037747 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.037710    2567 generic.go:356] "Generic (PLEG): container finished" podID="c0f9b192-ba1b-431c-80e6-d7f35c95ecb8" containerID="8c70550af69534debe632133d61b9340d4b7a851a92b51e8166839c56c9de5a1" exitCode=0
Sep 07 23:02:45.037920 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.037758    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl"
Sep 07 23:02:45.037920 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.037794    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl" event={"ID":"c0f9b192-ba1b-431c-80e6-d7f35c95ecb8","Type":"ContainerDied","Data":"8c70550af69534debe632133d61b9340d4b7a851a92b51e8166839c56c9de5a1"}
Sep 07 23:02:45.037920 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.037847    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl" event={"ID":"c0f9b192-ba1b-431c-80e6-d7f35c95ecb8","Type":"ContainerDied","Data":"5a8caeb69d23629f36217104e47122a550eb1c358ee1935e8720dc1054b904ab"}
Sep 07 23:02:45.037920 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.037875    2567 scope.go:117] "RemoveContainer" containerID="8c70550af69534debe632133d61b9340d4b7a851a92b51e8166839c56c9de5a1"
Sep 07 23:02:45.039173 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.039151    2567 generic.go:356] "Generic (PLEG): container finished" podID="6ab5cdb6-fa92-4787-ba43-2dbaaebfb331" containerID="2ff0e7f27b12fa178eacd801ce10786cf6af137424e431abe32ced2f53d3eff5" exitCode=0
Sep 07 23:02:45.039294 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.039206    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr"
Sep 07 23:02:45.039294 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.039218    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr" event={"ID":"6ab5cdb6-fa92-4787-ba43-2dbaaebfb331","Type":"ContainerDied","Data":"2ff0e7f27b12fa178eacd801ce10786cf6af137424e431abe32ced2f53d3eff5"}
Sep 07 23:02:45.039294 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.039277    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr" event={"ID":"6ab5cdb6-fa92-4787-ba43-2dbaaebfb331","Type":"ContainerDied","Data":"0de68ed1473f6906ce1b48e6a5869e00d3a722fb0e56def3bef0c7ff3808d4dd"}
Sep 07 23:02:45.046984 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.046960    2567 scope.go:117] "RemoveContainer" containerID="8c70550af69534debe632133d61b9340d4b7a851a92b51e8166839c56c9de5a1"
Sep 07 23:02:45.047326 ip-10-0-138-99 kubenswrapper[2567]: E0907 23:02:45.047306    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c70550af69534debe632133d61b9340d4b7a851a92b51e8166839c56c9de5a1\": container with ID starting with 8c70550af69534debe632133d61b9340d4b7a851a92b51e8166839c56c9de5a1 not found: ID does not exist" containerID="8c70550af69534debe632133d61b9340d4b7a851a92b51e8166839c56c9de5a1"
Sep 07 23:02:45.047400 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.047334    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c70550af69534debe632133d61b9340d4b7a851a92b51e8166839c56c9de5a1"} err="failed to get container status \"8c70550af69534debe632133d61b9340d4b7a851a92b51e8166839c56c9de5a1\": rpc error: code = NotFound desc = could not find container \"8c70550af69534debe632133d61b9340d4b7a851a92b51e8166839c56c9de5a1\": container with ID starting with 8c70550af69534debe632133d61b9340d4b7a851a92b51e8166839c56c9de5a1 not found: ID does not exist"
Sep 07 23:02:45.047400 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.047350    2567 scope.go:117] "RemoveContainer" containerID="2ff0e7f27b12fa178eacd801ce10786cf6af137424e431abe32ced2f53d3eff5"
Sep 07 23:02:45.055928 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.055911    2567 scope.go:117] "RemoveContainer" containerID="2ff0e7f27b12fa178eacd801ce10786cf6af137424e431abe32ced2f53d3eff5"
Sep 07 23:02:45.056216 ip-10-0-138-99 kubenswrapper[2567]: E0907 23:02:45.056198    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ff0e7f27b12fa178eacd801ce10786cf6af137424e431abe32ced2f53d3eff5\": container with ID starting with 2ff0e7f27b12fa178eacd801ce10786cf6af137424e431abe32ced2f53d3eff5 not found: ID does not exist" containerID="2ff0e7f27b12fa178eacd801ce10786cf6af137424e431abe32ced2f53d3eff5"
Sep 07 23:02:45.056318 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.056228    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ff0e7f27b12fa178eacd801ce10786cf6af137424e431abe32ced2f53d3eff5"} err="failed to get container status \"2ff0e7f27b12fa178eacd801ce10786cf6af137424e431abe32ced2f53d3eff5\": rpc error: code = NotFound desc = could not find container \"2ff0e7f27b12fa178eacd801ce10786cf6af137424e431abe32ced2f53d3eff5\": container with ID starting with 2ff0e7f27b12fa178eacd801ce10786cf6af137424e431abe32ced2f53d3eff5 not found: ID does not exist"
Sep 07 23:02:45.062166 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.062136    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl"]
Sep 07 23:02:45.067881 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.067855    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7823cc-6769bdcddb-k5lsl"]
Sep 07 23:02:45.076883 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.076856    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr"]
Sep 07 23:02:45.081858 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.081833    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7823cc-545bd44949-sknxr"]
Sep 07 23:02:45.466218 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.466183    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ab5cdb6-fa92-4787-ba43-2dbaaebfb331" path="/var/lib/kubelet/pods/6ab5cdb6-fa92-4787-ba43-2dbaaebfb331/volumes"
Sep 07 23:02:45.466615 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:45.466566    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0f9b192-ba1b-431c-80e6-d7f35c95ecb8" path="/var/lib/kubelet/pods/c0f9b192-ba1b-431c-80e6-d7f35c95ecb8/volumes"
Sep 07 23:02:53.637352 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:53.637314    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65f4bf68db-w46tm"]
Sep 07 23:02:53.637790 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:53.637510    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-65f4bf68db-w46tm" podUID="d760100e-6635-4866-b622-3f84c7a52756" containerName="manager" containerID="cri-o://8c29714df366b3dd1030e979d563999527e0cc66b5698e9bc9f865c1eb16a730" gracePeriod=30
Sep 07 23:02:53.887602 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:53.887540    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f4bf68db-w46tm"
Sep 07 23:02:54.037923 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:54.037881    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d760100e-6635-4866-b622-3f84c7a52756-metrics-certs\") pod \"d760100e-6635-4866-b622-3f84c7a52756\" (UID: \"d760100e-6635-4866-b622-3f84c7a52756\") "
Sep 07 23:02:54.038133 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:54.037999    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfllk\" (UniqueName: \"kubernetes.io/projected/d760100e-6635-4866-b622-3f84c7a52756-kube-api-access-vfllk\") pod \"d760100e-6635-4866-b622-3f84c7a52756\" (UID: \"d760100e-6635-4866-b622-3f84c7a52756\") "
Sep 07 23:02:54.038133 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:54.038031    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d760100e-6635-4866-b622-3f84c7a52756-webhook-cert\") pod \"d760100e-6635-4866-b622-3f84c7a52756\" (UID: \"d760100e-6635-4866-b622-3f84c7a52756\") "
Sep 07 23:02:54.040093 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:54.040059    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d760100e-6635-4866-b622-3f84c7a52756-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d760100e-6635-4866-b622-3f84c7a52756" (UID: "d760100e-6635-4866-b622-3f84c7a52756"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:02:54.040093 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:54.040086    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d760100e-6635-4866-b622-3f84c7a52756-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d760100e-6635-4866-b622-3f84c7a52756" (UID: "d760100e-6635-4866-b622-3f84c7a52756"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:02:54.040295 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:54.040147    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d760100e-6635-4866-b622-3f84c7a52756-kube-api-access-vfllk" (OuterVolumeSpecName: "kube-api-access-vfllk") pod "d760100e-6635-4866-b622-3f84c7a52756" (UID: "d760100e-6635-4866-b622-3f84c7a52756"). InnerVolumeSpecName "kube-api-access-vfllk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:02:54.083587 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:54.083551    2567 generic.go:356] "Generic (PLEG): container finished" podID="d760100e-6635-4866-b622-3f84c7a52756" containerID="8c29714df366b3dd1030e979d563999527e0cc66b5698e9bc9f865c1eb16a730" exitCode=0
Sep 07 23:02:54.083770 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:54.083595    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f4bf68db-w46tm"
Sep 07 23:02:54.083770 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:54.083637    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f4bf68db-w46tm" event={"ID":"d760100e-6635-4866-b622-3f84c7a52756","Type":"ContainerDied","Data":"8c29714df366b3dd1030e979d563999527e0cc66b5698e9bc9f865c1eb16a730"}
Sep 07 23:02:54.083770 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:54.083693    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f4bf68db-w46tm" event={"ID":"d760100e-6635-4866-b622-3f84c7a52756","Type":"ContainerDied","Data":"ac8a862bc1a995e67db499fdc46666a3298d69384655e0209ddf3428f79df36c"}
Sep 07 23:02:54.083770 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:54.083717    2567 scope.go:117] "RemoveContainer" containerID="8c29714df366b3dd1030e979d563999527e0cc66b5698e9bc9f865c1eb16a730"
Sep 07 23:02:54.093606 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:54.093584    2567 scope.go:117] "RemoveContainer" containerID="8c29714df366b3dd1030e979d563999527e0cc66b5698e9bc9f865c1eb16a730"
Sep 07 23:02:54.093931 ip-10-0-138-99 kubenswrapper[2567]: E0907 23:02:54.093909    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c29714df366b3dd1030e979d563999527e0cc66b5698e9bc9f865c1eb16a730\": container with ID starting with 8c29714df366b3dd1030e979d563999527e0cc66b5698e9bc9f865c1eb16a730 not found: ID does not exist" containerID="8c29714df366b3dd1030e979d563999527e0cc66b5698e9bc9f865c1eb16a730"
Sep 07 23:02:54.093985 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:54.093942    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c29714df366b3dd1030e979d563999527e0cc66b5698e9bc9f865c1eb16a730"} err="failed to get container status \"8c29714df366b3dd1030e979d563999527e0cc66b5698e9bc9f865c1eb16a730\": rpc error: code = NotFound desc = could not find container \"8c29714df366b3dd1030e979d563999527e0cc66b5698e9bc9f865c1eb16a730\": container with ID starting with 8c29714df366b3dd1030e979d563999527e0cc66b5698e9bc9f865c1eb16a730 not found: ID does not exist"
Sep 07 23:02:54.104671 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:54.104634    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65f4bf68db-w46tm"]
Sep 07 23:02:54.110401 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:54.110372    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65f4bf68db-w46tm"]
Sep 07 23:02:54.139329 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:54.139229    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d760100e-6635-4866-b622-3f84c7a52756-metrics-certs\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:02:54.139329 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:54.139279    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfllk\" (UniqueName: \"kubernetes.io/projected/d760100e-6635-4866-b622-3f84c7a52756-kube-api-access-vfllk\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:02:54.139329 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:54.139292    2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d760100e-6635-4866-b622-3f84c7a52756-webhook-cert\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:02:55.465567 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:02:55.465535    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d760100e-6635-4866-b622-3f84c7a52756" path="/var/lib/kubelet/pods/d760100e-6635-4866-b622-3f84c7a52756/volumes"
Sep 07 23:03:54.885968 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.885927    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65f4bf68db-bxq25"]
Sep 07 23:03:54.886453 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886338    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27cb8308-22b9-49b9-bea5-95163b41a409" containerName="maas-api"
Sep 07 23:03:54.886453 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886351    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="27cb8308-22b9-49b9-bea5-95163b41a409" containerName="maas-api"
Sep 07 23:03:54.886453 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886365    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d760100e-6635-4866-b622-3f84c7a52756" containerName="manager"
Sep 07 23:03:54.886453 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886374    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d760100e-6635-4866-b622-3f84c7a52756" containerName="manager"
Sep 07 23:03:54.886453 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886384    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f8b327c-b8ad-4bec-b88b-4e66ab32462b" containerName="payload-processing"
Sep 07 23:03:54.886453 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886390    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f8b327c-b8ad-4bec-b88b-4e66ab32462b" containerName="payload-processing"
Sep 07 23:03:54.886453 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886399    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c6c3708-549a-4e52-87fa-8ae773a62bdd" containerName="payload-pre-processing"
Sep 07 23:03:54.886453 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886406    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c6c3708-549a-4e52-87fa-8ae773a62bdd" containerName="payload-pre-processing"
Sep 07 23:03:54.886453 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886413    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab8df4d3-0a6f-42cd-8875-331b60960df6" containerName="payload-processing"
Sep 07 23:03:54.886453 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886419    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab8df4d3-0a6f-42cd-8875-331b60960df6" containerName="payload-processing"
Sep 07 23:03:54.886453 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886429    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ab5cdb6-fa92-4787-ba43-2dbaaebfb331" containerName="payload-processing"
Sep 07 23:03:54.886453 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886434    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ab5cdb6-fa92-4787-ba43-2dbaaebfb331" containerName="payload-processing"
Sep 07 23:03:54.886453 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886446    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68079a96-828c-4ccd-9b3f-a6c25f1d3297" containerName="payload-processing"
Sep 07 23:03:54.886453 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886454    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="68079a96-828c-4ccd-9b3f-a6c25f1d3297" containerName="payload-processing"
Sep 07 23:03:54.886453 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886462    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0f9b192-ba1b-431c-80e6-d7f35c95ecb8" containerName="payload-pre-processing"
Sep 07 23:03:54.886966 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886467    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0f9b192-ba1b-431c-80e6-d7f35c95ecb8" containerName="payload-pre-processing"
Sep 07 23:03:54.886966 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886479    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8d71988-7a6a-4bb8-b2a0-0e9697aabb55" containerName="payload-pre-processing"
Sep 07 23:03:54.886966 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886484    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8d71988-7a6a-4bb8-b2a0-0e9697aabb55" containerName="payload-pre-processing"
Sep 07 23:03:54.886966 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886546    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0f9b192-ba1b-431c-80e6-d7f35c95ecb8" containerName="payload-pre-processing"
Sep 07 23:03:54.886966 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886556    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ab5cdb6-fa92-4787-ba43-2dbaaebfb331" containerName="payload-processing"
Sep 07 23:03:54.886966 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886561    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="27cb8308-22b9-49b9-bea5-95163b41a409" containerName="maas-api"
Sep 07 23:03:54.886966 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886568    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c6c3708-549a-4e52-87fa-8ae773a62bdd" containerName="payload-pre-processing"
Sep 07 23:03:54.886966 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886576    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="68079a96-828c-4ccd-9b3f-a6c25f1d3297" containerName="payload-processing"
Sep 07 23:03:54.886966 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886582    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f8b327c-b8ad-4bec-b88b-4e66ab32462b" containerName="payload-processing"
Sep 07 23:03:54.886966 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886589    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d760100e-6635-4866-b622-3f84c7a52756" containerName="manager"
Sep 07 23:03:54.886966 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886594    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8d71988-7a6a-4bb8-b2a0-0e9697aabb55" containerName="payload-pre-processing"
Sep 07 23:03:54.886966 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.886602    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab8df4d3-0a6f-42cd-8875-331b60960df6" containerName="payload-processing"
Sep 07 23:03:54.889727 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.889708    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f4bf68db-bxq25"
Sep 07 23:03:54.892829 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.892803    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 07 23:03:54.892829 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.892808    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 07 23:03:54.893061 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.892843    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 07 23:03:54.893061 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.892847    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9hxct\""
Sep 07 23:03:54.900286 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.900255    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f4bf68db-bxq25"]
Sep 07 23:03:54.902908 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.902879    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22e2a9d2-741a-457f-a635-1f679073b32d-webhook-cert\") pod \"maas-controller-65f4bf68db-bxq25\" (UID: \"22e2a9d2-741a-457f-a635-1f679073b32d\") " pod="opendatahub/maas-controller-65f4bf68db-bxq25"
Sep 07 23:03:54.903050 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.902971    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22e2a9d2-741a-457f-a635-1f679073b32d-metrics-certs\") pod \"maas-controller-65f4bf68db-bxq25\" (UID: \"22e2a9d2-741a-457f-a635-1f679073b32d\") " pod="opendatahub/maas-controller-65f4bf68db-bxq25"
Sep 07 23:03:54.903122 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:54.903058    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhgb7\" (UniqueName: \"kubernetes.io/projected/22e2a9d2-741a-457f-a635-1f679073b32d-kube-api-access-nhgb7\") pod \"maas-controller-65f4bf68db-bxq25\" (UID: \"22e2a9d2-741a-457f-a635-1f679073b32d\") " pod="opendatahub/maas-controller-65f4bf68db-bxq25"
Sep 07 23:03:55.003730 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:55.003682    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22e2a9d2-741a-457f-a635-1f679073b32d-webhook-cert\") pod \"maas-controller-65f4bf68db-bxq25\" (UID: \"22e2a9d2-741a-457f-a635-1f679073b32d\") " pod="opendatahub/maas-controller-65f4bf68db-bxq25"
Sep 07 23:03:55.003924 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:55.003754    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22e2a9d2-741a-457f-a635-1f679073b32d-metrics-certs\") pod \"maas-controller-65f4bf68db-bxq25\" (UID: \"22e2a9d2-741a-457f-a635-1f679073b32d\") " pod="opendatahub/maas-controller-65f4bf68db-bxq25"
Sep 07 23:03:55.003924 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:55.003785    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhgb7\" (UniqueName: \"kubernetes.io/projected/22e2a9d2-741a-457f-a635-1f679073b32d-kube-api-access-nhgb7\") pod \"maas-controller-65f4bf68db-bxq25\" (UID: \"22e2a9d2-741a-457f-a635-1f679073b32d\") " pod="opendatahub/maas-controller-65f4bf68db-bxq25"
Sep 07 23:03:55.006328 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:55.006301    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22e2a9d2-741a-457f-a635-1f679073b32d-webhook-cert\") pod \"maas-controller-65f4bf68db-bxq25\" (UID: \"22e2a9d2-741a-457f-a635-1f679073b32d\") " pod="opendatahub/maas-controller-65f4bf68db-bxq25"
Sep 07 23:03:55.006450 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:55.006302    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22e2a9d2-741a-457f-a635-1f679073b32d-metrics-certs\") pod \"maas-controller-65f4bf68db-bxq25\" (UID: \"22e2a9d2-741a-457f-a635-1f679073b32d\") " pod="opendatahub/maas-controller-65f4bf68db-bxq25"
Sep 07 23:03:55.010876 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:55.010839    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhgb7\" (UniqueName: \"kubernetes.io/projected/22e2a9d2-741a-457f-a635-1f679073b32d-kube-api-access-nhgb7\") pod \"maas-controller-65f4bf68db-bxq25\" (UID: \"22e2a9d2-741a-457f-a635-1f679073b32d\") " pod="opendatahub/maas-controller-65f4bf68db-bxq25"
Sep 07 23:03:55.201725 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:55.201629    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f4bf68db-bxq25"
Sep 07 23:03:55.474314 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:55.474283    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f4bf68db-bxq25"]
Sep 07 23:03:55.477272 ip-10-0-138-99 kubenswrapper[2567]: W0907 23:03:55.477219    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e2a9d2_741a_457f_a635_1f679073b32d.slice/crio-5ca6911f2426fea0701334e2a7cac5a6ebef54620a47dd95fb3764d882176ae4 WatchSource:0}: Error finding container 5ca6911f2426fea0701334e2a7cac5a6ebef54620a47dd95fb3764d882176ae4: Status 404 returned error can't find the container with id 5ca6911f2426fea0701334e2a7cac5a6ebef54620a47dd95fb3764d882176ae4
Sep 07 23:03:55.478862 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:55.478839    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 23:03:56.388283 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:56.388226    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f4bf68db-bxq25" event={"ID":"22e2a9d2-741a-457f-a635-1f679073b32d","Type":"ContainerStarted","Data":"e45835837a0ff7dfcb1c3abc58f0c0244c4747b0d1853b65f6f65adcf290923f"}
Sep 07 23:03:56.388677 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:56.388307    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f4bf68db-bxq25" event={"ID":"22e2a9d2-741a-457f-a635-1f679073b32d","Type":"ContainerStarted","Data":"5ca6911f2426fea0701334e2a7cac5a6ebef54620a47dd95fb3764d882176ae4"}
Sep 07 23:03:56.388677 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:56.388384    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65f4bf68db-bxq25"
Sep 07 23:03:56.403027 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:03:56.402965    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65f4bf68db-bxq25" podStartSLOduration=1.984419881 podStartE2EDuration="2.40294602s" totalImagesPullingTime="418.526139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 23:03:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:03:56.401944263 +0000 UTC m=+1919.458777226" watchObservedRunningTime="2026-09-07 23:03:56.40294602 +0000 UTC m=+1919.459778981"
Sep 07 23:04:07.396369 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:07.396332    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65f4bf68db-bxq25"
Sep 07 23:04:16.698493 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:16.698449    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65f4bf68db-bxq25"]
Sep 07 23:04:16.698969 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:16.698735    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-65f4bf68db-bxq25" podUID="22e2a9d2-741a-457f-a635-1f679073b32d" containerName="manager" containerID="cri-o://e45835837a0ff7dfcb1c3abc58f0c0244c4747b0d1853b65f6f65adcf290923f" gracePeriod=30
Sep 07 23:04:16.944023 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:16.943997    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f4bf68db-bxq25"
Sep 07 23:04:17.009820 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:17.009725    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22e2a9d2-741a-457f-a635-1f679073b32d-metrics-certs\") pod \"22e2a9d2-741a-457f-a635-1f679073b32d\" (UID: \"22e2a9d2-741a-457f-a635-1f679073b32d\") "
Sep 07 23:04:17.009992 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:17.009857    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhgb7\" (UniqueName: \"kubernetes.io/projected/22e2a9d2-741a-457f-a635-1f679073b32d-kube-api-access-nhgb7\") pod \"22e2a9d2-741a-457f-a635-1f679073b32d\" (UID: \"22e2a9d2-741a-457f-a635-1f679073b32d\") "
Sep 07 23:04:17.009992 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:17.009930    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22e2a9d2-741a-457f-a635-1f679073b32d-webhook-cert\") pod \"22e2a9d2-741a-457f-a635-1f679073b32d\" (UID: \"22e2a9d2-741a-457f-a635-1f679073b32d\") "
Sep 07 23:04:17.011975 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:17.011938    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e2a9d2-741a-457f-a635-1f679073b32d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "22e2a9d2-741a-457f-a635-1f679073b32d" (UID: "22e2a9d2-741a-457f-a635-1f679073b32d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:04:17.012104 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:17.012002    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e2a9d2-741a-457f-a635-1f679073b32d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "22e2a9d2-741a-457f-a635-1f679073b32d" (UID: "22e2a9d2-741a-457f-a635-1f679073b32d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:04:17.012104 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:17.012074    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e2a9d2-741a-457f-a635-1f679073b32d-kube-api-access-nhgb7" (OuterVolumeSpecName: "kube-api-access-nhgb7") pod "22e2a9d2-741a-457f-a635-1f679073b32d" (UID: "22e2a9d2-741a-457f-a635-1f679073b32d"). InnerVolumeSpecName "kube-api-access-nhgb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:04:17.111001 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:17.110963    2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22e2a9d2-741a-457f-a635-1f679073b32d-webhook-cert\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:04:17.111001 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:17.110994    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22e2a9d2-741a-457f-a635-1f679073b32d-metrics-certs\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:04:17.111001 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:17.111004    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhgb7\" (UniqueName: \"kubernetes.io/projected/22e2a9d2-741a-457f-a635-1f679073b32d-kube-api-access-nhgb7\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:04:17.492798 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:17.492764    2567 generic.go:356] "Generic (PLEG): container finished" podID="22e2a9d2-741a-457f-a635-1f679073b32d" containerID="e45835837a0ff7dfcb1c3abc58f0c0244c4747b0d1853b65f6f65adcf290923f" exitCode=0
Sep 07 23:04:17.493001 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:17.492816    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f4bf68db-bxq25"
Sep 07 23:04:17.493001 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:17.492835    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f4bf68db-bxq25" event={"ID":"22e2a9d2-741a-457f-a635-1f679073b32d","Type":"ContainerDied","Data":"e45835837a0ff7dfcb1c3abc58f0c0244c4747b0d1853b65f6f65adcf290923f"}
Sep 07 23:04:17.493001 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:17.492878    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f4bf68db-bxq25" event={"ID":"22e2a9d2-741a-457f-a635-1f679073b32d","Type":"ContainerDied","Data":"5ca6911f2426fea0701334e2a7cac5a6ebef54620a47dd95fb3764d882176ae4"}
Sep 07 23:04:17.493001 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:17.492900    2567 scope.go:117] "RemoveContainer" containerID="e45835837a0ff7dfcb1c3abc58f0c0244c4747b0d1853b65f6f65adcf290923f"
Sep 07 23:04:17.502166 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:17.502142    2567 scope.go:117] "RemoveContainer" containerID="e45835837a0ff7dfcb1c3abc58f0c0244c4747b0d1853b65f6f65adcf290923f"
Sep 07 23:04:17.502511 ip-10-0-138-99 kubenswrapper[2567]: E0907 23:04:17.502489    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e45835837a0ff7dfcb1c3abc58f0c0244c4747b0d1853b65f6f65adcf290923f\": container with ID starting with e45835837a0ff7dfcb1c3abc58f0c0244c4747b0d1853b65f6f65adcf290923f not found: ID does not exist" containerID="e45835837a0ff7dfcb1c3abc58f0c0244c4747b0d1853b65f6f65adcf290923f"
Sep 07 23:04:17.502602 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:17.502520    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e45835837a0ff7dfcb1c3abc58f0c0244c4747b0d1853b65f6f65adcf290923f"} err="failed to get container status \"e45835837a0ff7dfcb1c3abc58f0c0244c4747b0d1853b65f6f65adcf290923f\": rpc error: code = NotFound desc = could not find container \"e45835837a0ff7dfcb1c3abc58f0c0244c4747b0d1853b65f6f65adcf290923f\": container with ID starting with e45835837a0ff7dfcb1c3abc58f0c0244c4747b0d1853b65f6f65adcf290923f not found: ID does not exist"
Sep 07 23:04:17.509129 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:17.509094    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65f4bf68db-bxq25"]
Sep 07 23:04:17.512829 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:17.512802    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65f4bf68db-bxq25"]
Sep 07 23:04:18.258120 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.258080    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65f4bf68db-vfwnw"]
Sep 07 23:04:18.258746 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.258669    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22e2a9d2-741a-457f-a635-1f679073b32d" containerName="manager"
Sep 07 23:04:18.258746 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.258691    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="22e2a9d2-741a-457f-a635-1f679073b32d" containerName="manager"
Sep 07 23:04:18.258871 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.258792    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="22e2a9d2-741a-457f-a635-1f679073b32d" containerName="manager"
Sep 07 23:04:18.263026 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.263001    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f4bf68db-vfwnw"
Sep 07 23:04:18.266106 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.266072    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 07 23:04:18.266106 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.266101    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 07 23:04:18.266350 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.266116    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 07 23:04:18.266350 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.266115    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9hxct\""
Sep 07 23:04:18.271662 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.271637    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f4bf68db-vfwnw"]
Sep 07 23:04:18.424827 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.424787    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6bv6\" (UniqueName: \"kubernetes.io/projected/c757c7f9-da23-43f1-821a-7b95b863ae71-kube-api-access-b6bv6\") pod \"maas-controller-65f4bf68db-vfwnw\" (UID: \"c757c7f9-da23-43f1-821a-7b95b863ae71\") " pod="opendatahub/maas-controller-65f4bf68db-vfwnw"
Sep 07 23:04:18.424827 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.424832    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c757c7f9-da23-43f1-821a-7b95b863ae71-metrics-certs\") pod \"maas-controller-65f4bf68db-vfwnw\" (UID: \"c757c7f9-da23-43f1-821a-7b95b863ae71\") " pod="opendatahub/maas-controller-65f4bf68db-vfwnw"
Sep 07 23:04:18.425053 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.424869    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c757c7f9-da23-43f1-821a-7b95b863ae71-webhook-cert\") pod \"maas-controller-65f4bf68db-vfwnw\" (UID: \"c757c7f9-da23-43f1-821a-7b95b863ae71\") " pod="opendatahub/maas-controller-65f4bf68db-vfwnw"
Sep 07 23:04:18.525539 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.525445    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6bv6\" (UniqueName: \"kubernetes.io/projected/c757c7f9-da23-43f1-821a-7b95b863ae71-kube-api-access-b6bv6\") pod \"maas-controller-65f4bf68db-vfwnw\" (UID: \"c757c7f9-da23-43f1-821a-7b95b863ae71\") " pod="opendatahub/maas-controller-65f4bf68db-vfwnw"
Sep 07 23:04:18.525539 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.525492    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c757c7f9-da23-43f1-821a-7b95b863ae71-metrics-certs\") pod \"maas-controller-65f4bf68db-vfwnw\" (UID: \"c757c7f9-da23-43f1-821a-7b95b863ae71\") " pod="opendatahub/maas-controller-65f4bf68db-vfwnw"
Sep 07 23:04:18.525539 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.525529    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c757c7f9-da23-43f1-821a-7b95b863ae71-webhook-cert\") pod \"maas-controller-65f4bf68db-vfwnw\" (UID: \"c757c7f9-da23-43f1-821a-7b95b863ae71\") " pod="opendatahub/maas-controller-65f4bf68db-vfwnw"
Sep 07 23:04:18.528060 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.528023    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c757c7f9-da23-43f1-821a-7b95b863ae71-metrics-certs\") pod \"maas-controller-65f4bf68db-vfwnw\" (UID: \"c757c7f9-da23-43f1-821a-7b95b863ae71\") " pod="opendatahub/maas-controller-65f4bf68db-vfwnw"
Sep 07 23:04:18.528189 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.528060    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c757c7f9-da23-43f1-821a-7b95b863ae71-webhook-cert\") pod \"maas-controller-65f4bf68db-vfwnw\" (UID: \"c757c7f9-da23-43f1-821a-7b95b863ae71\") " pod="opendatahub/maas-controller-65f4bf68db-vfwnw"
Sep 07 23:04:18.532651 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.532625    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6bv6\" (UniqueName: \"kubernetes.io/projected/c757c7f9-da23-43f1-821a-7b95b863ae71-kube-api-access-b6bv6\") pod \"maas-controller-65f4bf68db-vfwnw\" (UID: \"c757c7f9-da23-43f1-821a-7b95b863ae71\") " pod="opendatahub/maas-controller-65f4bf68db-vfwnw"
Sep 07 23:04:18.575328 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.575289    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f4bf68db-vfwnw"
Sep 07 23:04:18.848863 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:18.848836    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f4bf68db-vfwnw"]
Sep 07 23:04:18.851056 ip-10-0-138-99 kubenswrapper[2567]: W0907 23:04:18.851027    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc757c7f9_da23_43f1_821a_7b95b863ae71.slice/crio-a8e7f1074ca41a4a509c6ab1712f4d31736031d636f69015f6eb975768903789 WatchSource:0}: Error finding container a8e7f1074ca41a4a509c6ab1712f4d31736031d636f69015f6eb975768903789: Status 404 returned error can't find the container with id a8e7f1074ca41a4a509c6ab1712f4d31736031d636f69015f6eb975768903789
Sep 07 23:04:19.472685 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:19.472644    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22e2a9d2-741a-457f-a635-1f679073b32d" path="/var/lib/kubelet/pods/22e2a9d2-741a-457f-a635-1f679073b32d/volumes"
Sep 07 23:04:19.505158 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:19.505122    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f4bf68db-vfwnw" event={"ID":"c757c7f9-da23-43f1-821a-7b95b863ae71","Type":"ContainerStarted","Data":"6e73d974696d73b30e5d2a6d0d0270021ba8619729d80744d7624628fd330453"}
Sep 07 23:04:19.505359 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:19.505174    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f4bf68db-vfwnw" event={"ID":"c757c7f9-da23-43f1-821a-7b95b863ae71","Type":"ContainerStarted","Data":"a8e7f1074ca41a4a509c6ab1712f4d31736031d636f69015f6eb975768903789"}
Sep 07 23:04:19.505418 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:19.505375    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65f4bf68db-vfwnw"
Sep 07 23:04:19.520941 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:19.520883    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65f4bf68db-vfwnw" podStartSLOduration=1.099275175 podStartE2EDuration="1.520864764s" totalImagesPullingTime="421.589589ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 23:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:04:19.519289879 +0000 UTC m=+1942.576122839" watchObservedRunningTime="2026-09-07 23:04:19.520864764 +0000 UTC m=+1942.577697759"
Sep 07 23:04:30.512549 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:04:30.512503    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65f4bf68db-vfwnw"
Sep 07 23:06:57.574205 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:06:57.574173    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/2.log"
Sep 07 23:06:57.583183 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:06:57.583152    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/2.log"
Sep 07 23:07:04.271599 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.271553    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"]
Sep 07 23:07:04.275143 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.275115    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.277744 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.277661    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-bc653fb1-kserve-self-signed-certs\""
Sep 07 23:07:04.285779 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.285743    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"]
Sep 07 23:07:04.375652 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.375616    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-home\") pod \"e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") " pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.375832 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.375671    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9jtx\" (UniqueName: \"kubernetes.io/projected/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-kube-api-access-l9jtx\") pod \"e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") " pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.375832 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.375764    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-model-cache\") pod \"e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") " pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.375832 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.375828    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-tls-certs\") pod \"e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") " pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.375940 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.375850    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-dshm\") pod \"e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") " pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.375940 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.375926    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-tmp-dir\") pod \"e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") " pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.476970 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.476930    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-model-cache\") pod \"e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") " pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.476970 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.476973    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-tls-certs\") pod \"e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") " pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.477222 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.476991    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-dshm\") pod \"e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") " pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.477222 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.477036    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-tmp-dir\") pod \"e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") " pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.477222 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.477070    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-home\") pod \"e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") " pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.477222 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.477113    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9jtx\" (UniqueName: \"kubernetes.io/projected/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-kube-api-access-l9jtx\") pod \"e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") " pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.477489 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.477459    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-model-cache\") pod \"e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") " pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.477554 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.477528    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-tmp-dir\") pod \"e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") " pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.477612 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.477585    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-home\") pod \"e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") " pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.479395 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.479377    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-dshm\") pod \"e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") " pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.479759 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.479743    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-tls-certs\") pod \"e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") " pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.483893 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.483873    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9jtx\" (UniqueName: \"kubernetes.io/projected/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-kube-api-access-l9jtx\") pod \"e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") " pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.592893 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.592856    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:04.864015 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:04.863988    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"]
Sep 07 23:07:04.866641 ip-10-0-138-99 kubenswrapper[2567]: W0907 23:07:04.866608    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c648295_6cf4_4ced_94fc_a5e3a4f382ca.slice/crio-c8b37558b0e8a0f0bffd5fcd47cdb3c2861224c8278d862d3a075d3b2c1dda84 WatchSource:0}: Error finding container c8b37558b0e8a0f0bffd5fcd47cdb3c2861224c8278d862d3a075d3b2c1dda84: Status 404 returned error can't find the container with id c8b37558b0e8a0f0bffd5fcd47cdb3c2861224c8278d862d3a075d3b2c1dda84
Sep 07 23:07:05.321367 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:05.321333    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt" event={"ID":"0c648295-6cf4-4ced-94fc-a5e3a4f382ca","Type":"ContainerStarted","Data":"71d046f0ee98aab7310dfa4c8d9b9c5ed2ff5cccc70e5bcb2a304e34637fcec1"}
Sep 07 23:07:05.321736 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:05.321376    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt" event={"ID":"0c648295-6cf4-4ced-94fc-a5e3a4f382ca","Type":"ContainerStarted","Data":"c8b37558b0e8a0f0bffd5fcd47cdb3c2861224c8278d862d3a075d3b2c1dda84"}
Sep 07 23:07:05.339982 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:05.339924    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt" podStartSLOduration=1.339906956 podStartE2EDuration="1.339906956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 23:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:07:05.339654664 +0000 UTC m=+2108.396487622" watchObservedRunningTime="2026-09-07 23:07:05.339906956 +0000 UTC m=+2108.396739921"
Sep 07 23:07:14.593526 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:14.593486    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:14.594031 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:14.593544    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:14.605932 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:14.605903    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:15.380315 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:15.380228    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:07:46.295476 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:07:46.295438    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"]
Sep 07 23:08:01.316908 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.316860    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt" podUID="0c648295-6cf4-4ced-94fc-a5e3a4f382ca" containerName="main" containerID="cri-o://71d046f0ee98aab7310dfa4c8d9b9c5ed2ff5cccc70e5bcb2a304e34637fcec1" gracePeriod=45
Sep 07 23:08:01.579700 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.579628    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:08:01.602935 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.602896    2567 generic.go:356] "Generic (PLEG): container finished" podID="0c648295-6cf4-4ced-94fc-a5e3a4f382ca" containerID="71d046f0ee98aab7310dfa4c8d9b9c5ed2ff5cccc70e5bcb2a304e34637fcec1" exitCode=0
Sep 07 23:08:01.603095 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.602945    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"
Sep 07 23:08:01.603095 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.602946    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt" event={"ID":"0c648295-6cf4-4ced-94fc-a5e3a4f382ca","Type":"ContainerDied","Data":"71d046f0ee98aab7310dfa4c8d9b9c5ed2ff5cccc70e5bcb2a304e34637fcec1"}
Sep 07 23:08:01.603095 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.602987    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt" event={"ID":"0c648295-6cf4-4ced-94fc-a5e3a4f382ca","Type":"ContainerDied","Data":"c8b37558b0e8a0f0bffd5fcd47cdb3c2861224c8278d862d3a075d3b2c1dda84"}
Sep 07 23:08:01.603095 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.603010    2567 scope.go:117] "RemoveContainer" containerID="71d046f0ee98aab7310dfa4c8d9b9c5ed2ff5cccc70e5bcb2a304e34637fcec1"
Sep 07 23:08:01.613162 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.613139    2567 scope.go:117] "RemoveContainer" containerID="71d046f0ee98aab7310dfa4c8d9b9c5ed2ff5cccc70e5bcb2a304e34637fcec1"
Sep 07 23:08:01.613556 ip-10-0-138-99 kubenswrapper[2567]: E0907 23:08:01.613537    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d046f0ee98aab7310dfa4c8d9b9c5ed2ff5cccc70e5bcb2a304e34637fcec1\": container with ID starting with 71d046f0ee98aab7310dfa4c8d9b9c5ed2ff5cccc70e5bcb2a304e34637fcec1 not found: ID does not exist" containerID="71d046f0ee98aab7310dfa4c8d9b9c5ed2ff5cccc70e5bcb2a304e34637fcec1"
Sep 07 23:08:01.613631 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.613566    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d046f0ee98aab7310dfa4c8d9b9c5ed2ff5cccc70e5bcb2a304e34637fcec1"} err="failed to get container status \"71d046f0ee98aab7310dfa4c8d9b9c5ed2ff5cccc70e5bcb2a304e34637fcec1\": rpc error: code = NotFound desc = could not find container \"71d046f0ee98aab7310dfa4c8d9b9c5ed2ff5cccc70e5bcb2a304e34637fcec1\": container with ID starting with 71d046f0ee98aab7310dfa4c8d9b9c5ed2ff5cccc70e5bcb2a304e34637fcec1 not found: ID does not exist"
Sep 07 23:08:01.633957 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.633919    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9jtx\" (UniqueName: \"kubernetes.io/projected/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-kube-api-access-l9jtx\") pod \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") "
Sep 07 23:08:01.634114 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.633996    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-tmp-dir\") pod \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") "
Sep 07 23:08:01.634114 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.634042    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-model-cache\") pod \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") "
Sep 07 23:08:01.634114 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.634063    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-dshm\") pod \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") "
Sep 07 23:08:01.634114 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.634095    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-home\") pod \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") "
Sep 07 23:08:01.634333 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.634124    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-tls-certs\") pod \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\" (UID: \"0c648295-6cf4-4ced-94fc-a5e3a4f382ca\") "
Sep 07 23:08:01.634403 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.634340    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c648295-6cf4-4ced-94fc-a5e3a4f382ca" (UID: "0c648295-6cf4-4ced-94fc-a5e3a4f382ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:08:01.634403 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.634336    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c648295-6cf4-4ced-94fc-a5e3a4f382ca" (UID: "0c648295-6cf4-4ced-94fc-a5e3a4f382ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:08:01.634496 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.634417    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-home" (OuterVolumeSpecName: "home") pod "0c648295-6cf4-4ced-94fc-a5e3a4f382ca" (UID: "0c648295-6cf4-4ced-94fc-a5e3a4f382ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:08:01.636335 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.636307    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-kube-api-access-l9jtx" (OuterVolumeSpecName: "kube-api-access-l9jtx") pod "0c648295-6cf4-4ced-94fc-a5e3a4f382ca" (UID: "0c648295-6cf4-4ced-94fc-a5e3a4f382ca"). InnerVolumeSpecName "kube-api-access-l9jtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:08:01.636434 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.636372    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c648295-6cf4-4ced-94fc-a5e3a4f382ca" (UID: "0c648295-6cf4-4ced-94fc-a5e3a4f382ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:08:01.636434 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.636378    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-dshm" (OuterVolumeSpecName: "dshm") pod "0c648295-6cf4-4ced-94fc-a5e3a4f382ca" (UID: "0c648295-6cf4-4ced-94fc-a5e3a4f382ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:08:01.735001 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.734958    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-tmp-dir\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:08:01.735001 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.734991    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-model-cache\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:08:01.735001 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.735004    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-dshm\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:08:01.735001 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.735014    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-home\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:08:01.735309 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.735022    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-tls-certs\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:08:01.735309 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.735031    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9jtx\" (UniqueName: \"kubernetes.io/projected/0c648295-6cf4-4ced-94fc-a5e3a4f382ca-kube-api-access-l9jtx\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:08:01.928817 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.928776    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"]
Sep 07 23:08:01.929078 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:01.929063    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-bc653fb1-kserve-669449c54c-hlvzt"]
Sep 07 23:08:03.465311 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:03.465265    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c648295-6cf4-4ced-94fc-a5e3a4f382ca" path="/var/lib/kubelet/pods/0c648295-6cf4-4ced-94fc-a5e3a4f382ca/volumes"
Sep 07 23:08:04.382594 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.382562    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9"]
Sep 07 23:08:04.382939 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.382927    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c648295-6cf4-4ced-94fc-a5e3a4f382ca" containerName="main"
Sep 07 23:08:04.382985 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.382941    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c648295-6cf4-4ced-94fc-a5e3a4f382ca" containerName="main"
Sep 07 23:08:04.383021 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.383016    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c648295-6cf4-4ced-94fc-a5e3a4f382ca" containerName="main"
Sep 07 23:08:04.386351 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.386326    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9"
Sep 07 23:08:04.401957 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.401925    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9"]
Sep 07 23:08:04.463588 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.463547    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp929\" (UniqueName: \"kubernetes.io/projected/a65d167c-2d6e-4039-9091-6b848d348046-kube-api-access-rp929\") pod \"maas-api-6d9d8796ff-fwhw9\" (UID: \"a65d167c-2d6e-4039-9091-6b848d348046\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9"
Sep 07 23:08:04.463777 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.463598    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a65d167c-2d6e-4039-9091-6b848d348046-openshift-service-ca\") pod \"maas-api-6d9d8796ff-fwhw9\" (UID: \"a65d167c-2d6e-4039-9091-6b848d348046\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9"
Sep 07 23:08:04.463777 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.463661    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a65d167c-2d6e-4039-9091-6b848d348046-maas-api-tls\") pod \"maas-api-6d9d8796ff-fwhw9\" (UID: \"a65d167c-2d6e-4039-9091-6b848d348046\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9"
Sep 07 23:08:04.463777 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.463750    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a65d167c-2d6e-4039-9091-6b848d348046-metrics-certs\") pod \"maas-api-6d9d8796ff-fwhw9\" (UID: \"a65d167c-2d6e-4039-9091-6b848d348046\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9"
Sep 07 23:08:04.564997 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.564955    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a65d167c-2d6e-4039-9091-6b848d348046-metrics-certs\") pod \"maas-api-6d9d8796ff-fwhw9\" (UID: \"a65d167c-2d6e-4039-9091-6b848d348046\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9"
Sep 07 23:08:04.564997 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.564996    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp929\" (UniqueName: \"kubernetes.io/projected/a65d167c-2d6e-4039-9091-6b848d348046-kube-api-access-rp929\") pod \"maas-api-6d9d8796ff-fwhw9\" (UID: \"a65d167c-2d6e-4039-9091-6b848d348046\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9"
Sep 07 23:08:04.565559 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.565024    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a65d167c-2d6e-4039-9091-6b848d348046-openshift-service-ca\") pod \"maas-api-6d9d8796ff-fwhw9\" (UID: \"a65d167c-2d6e-4039-9091-6b848d348046\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9"
Sep 07 23:08:04.565559 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.565060    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a65d167c-2d6e-4039-9091-6b848d348046-maas-api-tls\") pod \"maas-api-6d9d8796ff-fwhw9\" (UID: \"a65d167c-2d6e-4039-9091-6b848d348046\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9"
Sep 07 23:08:04.565798 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.565773    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a65d167c-2d6e-4039-9091-6b848d348046-openshift-service-ca\") pod \"maas-api-6d9d8796ff-fwhw9\" (UID: \"a65d167c-2d6e-4039-9091-6b848d348046\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9"
Sep 07 23:08:04.567732 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.567700    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a65d167c-2d6e-4039-9091-6b848d348046-maas-api-tls\") pod \"maas-api-6d9d8796ff-fwhw9\" (UID: \"a65d167c-2d6e-4039-9091-6b848d348046\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9"
Sep 07 23:08:04.567732 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.567720    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a65d167c-2d6e-4039-9091-6b848d348046-metrics-certs\") pod \"maas-api-6d9d8796ff-fwhw9\" (UID: \"a65d167c-2d6e-4039-9091-6b848d348046\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9"
Sep 07 23:08:04.572196 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.572170    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp929\" (UniqueName: \"kubernetes.io/projected/a65d167c-2d6e-4039-9091-6b848d348046-kube-api-access-rp929\") pod \"maas-api-6d9d8796ff-fwhw9\" (UID: \"a65d167c-2d6e-4039-9091-6b848d348046\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9"
Sep 07 23:08:04.696496 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.696389    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9"
Sep 07 23:08:04.974728 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:04.974638    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9"]
Sep 07 23:08:04.978167 ip-10-0-138-99 kubenswrapper[2567]: W0907 23:08:04.978136    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65d167c_2d6e_4039_9091_6b848d348046.slice/crio-827385c9806fb43d36fd21de0e8726642843e92a1f8bbd3e502ac336cc93b0dc WatchSource:0}: Error finding container 827385c9806fb43d36fd21de0e8726642843e92a1f8bbd3e502ac336cc93b0dc: Status 404 returned error can't find the container with id 827385c9806fb43d36fd21de0e8726642843e92a1f8bbd3e502ac336cc93b0dc
Sep 07 23:08:05.627096 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:05.627002    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9" event={"ID":"a65d167c-2d6e-4039-9091-6b848d348046","Type":"ContainerStarted","Data":"032b57903520e7976e5255c7c57c10ea8158650538327860a33e9cb1d8a3568b"}
Sep 07 23:08:05.627096 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:05.627057    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9" event={"ID":"a65d167c-2d6e-4039-9091-6b848d348046","Type":"ContainerStarted","Data":"827385c9806fb43d36fd21de0e8726642843e92a1f8bbd3e502ac336cc93b0dc"}
Sep 07 23:08:05.627096 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:05.627085    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9"
Sep 07 23:08:05.642962 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:05.642904    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9" podStartSLOduration=1.248859662 podStartE2EDuration="1.642887205s" totalImagesPullingTime="394.027543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 23:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:08:05.640737921 +0000 UTC m=+2168.697570882" watchObservedRunningTime="2026-09-07 23:08:05.642887205 +0000 UTC m=+2168.699720165"
Sep 07 23:08:11.638195 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:11.638163    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-fwhw9"
Sep 07 23:08:19.442448 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:19.442410    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"]
Sep 07 23:08:19.443004 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:19.442624    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4" podUID="18379880-15f3-418c-b1b8-95a73b92aa7e" containerName="maas-api" containerID="cri-o://3c11ab287281ed1cd2bb40be786af10575599aad6073cde2661bd66d0ef12b68" gracePeriod=30
Sep 07 23:08:20.223517 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.223459    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4" podUID="18379880-15f3-418c-b1b8-95a73b92aa7e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.36:8443/health\": dial tcp 10.132.0.36:8443: connect: connection refused"
Sep 07 23:08:20.691403 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.691375    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 23:08:20.703981 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.703948    2567 generic.go:356] "Generic (PLEG): container finished" podID="18379880-15f3-418c-b1b8-95a73b92aa7e" containerID="3c11ab287281ed1cd2bb40be786af10575599aad6073cde2661bd66d0ef12b68" exitCode=0
Sep 07 23:08:20.704148 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.703996    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"
Sep 07 23:08:20.704148 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.704030    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4" event={"ID":"18379880-15f3-418c-b1b8-95a73b92aa7e","Type":"ContainerDied","Data":"3c11ab287281ed1cd2bb40be786af10575599aad6073cde2661bd66d0ef12b68"}
Sep 07 23:08:20.704148 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.704086    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4" event={"ID":"18379880-15f3-418c-b1b8-95a73b92aa7e","Type":"ContainerDied","Data":"9f1adc49f8561a4452daf49bac3ec857e53cc6cc58612bc4fee5646b60be3e9e"}
Sep 07 23:08:20.704148 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.704112    2567 scope.go:117] "RemoveContainer" containerID="3c11ab287281ed1cd2bb40be786af10575599aad6073cde2661bd66d0ef12b68"
Sep 07 23:08:20.716356 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.716329    2567 scope.go:117] "RemoveContainer" containerID="3c11ab287281ed1cd2bb40be786af10575599aad6073cde2661bd66d0ef12b68"
Sep 07 23:08:20.716753 ip-10-0-138-99 kubenswrapper[2567]: E0907 23:08:20.716732    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c11ab287281ed1cd2bb40be786af10575599aad6073cde2661bd66d0ef12b68\": container with ID starting with 3c11ab287281ed1cd2bb40be786af10575599aad6073cde2661bd66d0ef12b68 not found: ID does not exist" containerID="3c11ab287281ed1cd2bb40be786af10575599aad6073cde2661bd66d0ef12b68"
Sep 07 23:08:20.716808 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.716764    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c11ab287281ed1cd2bb40be786af10575599aad6073cde2661bd66d0ef12b68"} err="failed to get container status \"3c11ab287281ed1cd2bb40be786af10575599aad6073cde2661bd66d0ef12b68\": rpc error: code = NotFound desc = could not find container \"3c11ab287281ed1cd2bb40be786af10575599aad6073cde2661bd66d0ef12b68\": container with ID starting with 3c11ab287281ed1cd2bb40be786af10575599aad6073cde2661bd66d0ef12b68 not found: ID does not exist"
Sep 07 23:08:20.830520 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.830484    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18379880-15f3-418c-b1b8-95a73b92aa7e-maas-api-tls\") pod \"18379880-15f3-418c-b1b8-95a73b92aa7e\" (UID: \"18379880-15f3-418c-b1b8-95a73b92aa7e\") "
Sep 07 23:08:20.830705 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.830549    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vwx9\" (UniqueName: \"kubernetes.io/projected/18379880-15f3-418c-b1b8-95a73b92aa7e-kube-api-access-2vwx9\") pod \"18379880-15f3-418c-b1b8-95a73b92aa7e\" (UID: \"18379880-15f3-418c-b1b8-95a73b92aa7e\") "
Sep 07 23:08:20.830705 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.830624    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18379880-15f3-418c-b1b8-95a73b92aa7e-metrics-certs\") pod \"18379880-15f3-418c-b1b8-95a73b92aa7e\" (UID: \"18379880-15f3-418c-b1b8-95a73b92aa7e\") "
Sep 07 23:08:20.830705 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.830667    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18379880-15f3-418c-b1b8-95a73b92aa7e-openshift-service-ca\") pod \"18379880-15f3-418c-b1b8-95a73b92aa7e\" (UID: \"18379880-15f3-418c-b1b8-95a73b92aa7e\") "
Sep 07 23:08:20.831100 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.831070    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18379880-15f3-418c-b1b8-95a73b92aa7e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "18379880-15f3-418c-b1b8-95a73b92aa7e" (UID: "18379880-15f3-418c-b1b8-95a73b92aa7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:08:20.832983 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.832955    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18379880-15f3-418c-b1b8-95a73b92aa7e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "18379880-15f3-418c-b1b8-95a73b92aa7e" (UID: "18379880-15f3-418c-b1b8-95a73b92aa7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:08:20.833185 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.833167    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18379880-15f3-418c-b1b8-95a73b92aa7e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "18379880-15f3-418c-b1b8-95a73b92aa7e" (UID: "18379880-15f3-418c-b1b8-95a73b92aa7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:08:20.833255 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.833187    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18379880-15f3-418c-b1b8-95a73b92aa7e-kube-api-access-2vwx9" (OuterVolumeSpecName: "kube-api-access-2vwx9") pod "18379880-15f3-418c-b1b8-95a73b92aa7e" (UID: "18379880-15f3-418c-b1b8-95a73b92aa7e"). InnerVolumeSpecName "kube-api-access-2vwx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:08:20.932099 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.932056    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18379880-15f3-418c-b1b8-95a73b92aa7e-maas-api-tls\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:08:20.932099 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.932093    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vwx9\" (UniqueName: \"kubernetes.io/projected/18379880-15f3-418c-b1b8-95a73b92aa7e-kube-api-access-2vwx9\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:08:20.932099 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.932105    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18379880-15f3-418c-b1b8-95a73b92aa7e-metrics-certs\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:08:20.932407 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:20.932114    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18379880-15f3-418c-b1b8-95a73b92aa7e-openshift-service-ca\") on node \"ip-10-0-138-99.ec2.internal\" DevicePath \"\""
Sep 07 23:08:21.027550 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:21.027517    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"]
Sep 07 23:08:21.030988 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:21.030956    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7778d67-sgsz4"]
Sep 07 23:08:21.466388 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:21.466346    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18379880-15f3-418c-b1b8-95a73b92aa7e" path="/var/lib/kubelet/pods/18379880-15f3-418c-b1b8-95a73b92aa7e/volumes"
Sep 07 23:08:35.976788 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:35.976747    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84449f4bcf-njv47_745ea2bb-3e2f-4303-8f3d-35a7a7684cc4/authorino/0.log"
Sep 07 23:08:41.130575 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:41.130529    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65f4bf68db-vfwnw_c757c7f9-da23-43f1-821a-7b95b863ae71/manager/0.log"
Sep 07 23:08:41.569814 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:41.569775    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d648d744d-s9jnn_aee02241-2234-4d9e-a6c5-3ae7b3ad8d13/manager/0.log"
Sep 07 23:08:41.575189 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:41.575162    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d648d744d-s9jnn_aee02241-2234-4d9e-a6c5-3ae7b3ad8d13/copy-manifests/0.log"
Sep 07 23:08:42.010472 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:42.010393    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d9d8796ff-fwhw9_a65d167c-2d6e-4039-9091-6b848d348046/maas-api/0.log"
Sep 07 23:08:42.324894 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:42.324861    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29813700-mc5gl_719d805a-4232-4f00-b8de-c4fabae29c0d/cleanup/0.log"
Sep 07 23:08:42.425237 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:42.425204    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-tz2vq_341581c6-0c54-443b-a949-78b942f80c5b/postgres/0.log"
Sep 07 23:08:43.157768 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:43.157731    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84449f4bcf-njv47_745ea2bb-3e2f-4303-8f3d-35a7a7684cc4/authorino/0.log"
Sep 07 23:08:44.446170 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:44.446141    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68c66dfc89-vdgsz_f0f0ca3c-6829-439a-b38b-642d0b090c9a/kube-auth-proxy/0.log"
Sep 07 23:08:44.771158 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:44.771070    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-6vqtn_d66e33da-96fe-433d-9dd2-3e5f1ef9f021/payload-processing/0.log"
Sep 07 23:08:45.608950 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:45.608915    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvz8jb_b2187c27-3fc2-4c2d-b874-f1ec7bc72db4/main/0.log"
Sep 07 23:08:45.713334 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:45.713302    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-7px2s_ed3d1c60-be53-4d65-8d4d-9828f14b11ff/main/0.log"
Sep 07 23:08:53.582999 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:53.582966    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jd8ww_1e97b2da-120f-4a72-a6ea-907bcb72a13e/global-pull-secret-syncer/0.log"
Sep 07 23:08:53.686917 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:53.686884    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n6df9_5efbc01b-b5b9-40df-9ba9-71acdfd8be6e/konnectivity-agent/0.log"
Sep 07 23:08:53.733157 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:53.733120    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-99.ec2.internal_9c67f07165e3fd7605dfe27e9a9c951a/haproxy/0.log"
Sep 07 23:08:57.790081 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:57.790032    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84449f4bcf-njv47_745ea2bb-3e2f-4303-8f3d-35a7a7684cc4/authorino/0.log"
Sep 07 23:08:59.887792 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:59.887760    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_daff1dfd-5a2e-4aab-a65d-70d20d7edf49/alertmanager/0.log"
Sep 07 23:08:59.909117 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:59.909083    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_daff1dfd-5a2e-4aab-a65d-70d20d7edf49/config-reloader/0.log"
Sep 07 23:08:59.926198 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:59.926161    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_daff1dfd-5a2e-4aab-a65d-70d20d7edf49/kube-rbac-proxy-web/0.log"
Sep 07 23:08:59.943927 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:59.943881    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_daff1dfd-5a2e-4aab-a65d-70d20d7edf49/kube-rbac-proxy/0.log"
Sep 07 23:08:59.963905 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:59.963862    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_daff1dfd-5a2e-4aab-a65d-70d20d7edf49/kube-rbac-proxy-metric/0.log"
Sep 07 23:08:59.981779 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:08:59.981748    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_daff1dfd-5a2e-4aab-a65d-70d20d7edf49/prom-label-proxy/0.log"
Sep 07 23:09:00.002566 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:00.002476    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_daff1dfd-5a2e-4aab-a65d-70d20d7edf49/init-config-reloader/0.log"
Sep 07 23:09:00.044889 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:00.044848    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-28hz4_3cd41b43-6357-4ec5-b4b1-0cfc785ea58b/cluster-monitoring-operator/0.log"
Sep 07 23:09:00.146665 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:00.146631    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5f858c696c-n9mdv_bcb39ca7-73b6-4fe8-b5b0-7f2dc796a4bf/metrics-server/0.log"
Sep 07 23:09:00.198000 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:00.197974    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4kjc5_1c93c4ac-72e8-454d-9c80-9a19a4efbcfd/node-exporter/0.log"
Sep 07 23:09:00.215106 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:00.215076    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4kjc5_1c93c4ac-72e8-454d-9c80-9a19a4efbcfd/kube-rbac-proxy/0.log"
Sep 07 23:09:00.232667 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:00.232641    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4kjc5_1c93c4ac-72e8-454d-9c80-9a19a4efbcfd/init-textfile/0.log"
Sep 07 23:09:00.486991 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:00.486960    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_91025335-55b9-45b6-ad36-079659f69853/prometheus/0.log"
Sep 07 23:09:00.504528 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:00.504494    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_91025335-55b9-45b6-ad36-079659f69853/config-reloader/0.log"
Sep 07 23:09:00.523125 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:00.523098    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_91025335-55b9-45b6-ad36-079659f69853/thanos-sidecar/0.log"
Sep 07 23:09:00.540691 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:00.540663    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_91025335-55b9-45b6-ad36-079659f69853/kube-rbac-proxy-web/0.log"
Sep 07 23:09:00.558308 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:00.558227    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_91025335-55b9-45b6-ad36-079659f69853/kube-rbac-proxy/0.log"
Sep 07 23:09:00.575296 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:00.575266    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_91025335-55b9-45b6-ad36-079659f69853/kube-rbac-proxy-thanos/0.log"
Sep 07 23:09:00.593440 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:00.593401    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_91025335-55b9-45b6-ad36-079659f69853/init-config-reloader/0.log"
Sep 07 23:09:00.619858 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:00.619830    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-vjxnd_6f563137-3790-4f41-ae58-98dbe5b37caa/prometheus-operator/0.log"
Sep 07 23:09:00.633949 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:00.633921    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-vjxnd_6f563137-3790-4f41-ae58-98dbe5b37caa/kube-rbac-proxy/0.log"
Sep 07 23:09:00.685355 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:00.685328    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76cddc7965-qc5k6_1a794ce9-1039-494e-a794-ef7c45159bdf/telemeter-client/0.log"
Sep 07 23:09:00.704027 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:00.703999    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76cddc7965-qc5k6_1a794ce9-1039-494e-a794-ef7c45159bdf/reload/0.log"
Sep 07 23:09:00.724654 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:00.724624    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76cddc7965-qc5k6_1a794ce9-1039-494e-a794-ef7c45159bdf/kube-rbac-proxy/0.log"
Sep 07 23:09:03.249105 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:03.249073    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/2.log"
Sep 07 23:09:03.256765 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:03.256727    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nrwrg_d3159cd3-b8ca-403e-a8c2-687db53940ab/console-operator/3.log"
Sep 07 23:09:03.794744 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:03.794706    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-d6b77_87bdbb8b-6a3b-4a92-a68e-3ead40f9d615/download-server/0.log"
Sep 07 23:09:04.180825 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.180777    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"]
Sep 07 23:09:04.181172 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.181159    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18379880-15f3-418c-b1b8-95a73b92aa7e" containerName="maas-api"
Sep 07 23:09:04.181218 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.181173    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="18379880-15f3-418c-b1b8-95a73b92aa7e" containerName="maas-api"
Sep 07 23:09:04.181277 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.181256    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="18379880-15f3-418c-b1b8-95a73b92aa7e" containerName="maas-api"
Sep 07 23:09:04.184313 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.184291    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"
Sep 07 23:09:04.186598 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.186565    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8p8fk\"/\"kube-root-ca.crt\""
Sep 07 23:09:04.187428 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.187404    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8p8fk\"/\"default-dockercfg-ktzj4\""
Sep 07 23:09:04.187638 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.187415    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8p8fk\"/\"openshift-service-ca.crt\""
Sep 07 23:09:04.192967 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.192944    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"]
Sep 07 23:09:04.328599 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.328557    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e7303a3e-e627-451c-9261-e553595fb53c-proc\") pod \"perf-node-gather-daemonset-bb5kk\" (UID: \"e7303a3e-e627-451c-9261-e553595fb53c\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"
Sep 07 23:09:04.328985 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.328610    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7303a3e-e627-451c-9261-e553595fb53c-sys\") pod \"perf-node-gather-daemonset-bb5kk\" (UID: \"e7303a3e-e627-451c-9261-e553595fb53c\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"
Sep 07 23:09:04.328985 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.328659    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7303a3e-e627-451c-9261-e553595fb53c-lib-modules\") pod \"perf-node-gather-daemonset-bb5kk\" (UID: \"e7303a3e-e627-451c-9261-e553595fb53c\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"
Sep 07 23:09:04.328985 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.328700    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e7303a3e-e627-451c-9261-e553595fb53c-podres\") pod \"perf-node-gather-daemonset-bb5kk\" (UID: \"e7303a3e-e627-451c-9261-e553595fb53c\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"
Sep 07 23:09:04.328985 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.328736    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trrzz\" (UniqueName: \"kubernetes.io/projected/e7303a3e-e627-451c-9261-e553595fb53c-kube-api-access-trrzz\") pod \"perf-node-gather-daemonset-bb5kk\" (UID: \"e7303a3e-e627-451c-9261-e553595fb53c\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"
Sep 07 23:09:04.429320 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.429281    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e7303a3e-e627-451c-9261-e553595fb53c-proc\") pod \"perf-node-gather-daemonset-bb5kk\" (UID: \"e7303a3e-e627-451c-9261-e553595fb53c\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"
Sep 07 23:09:04.429485 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.429328    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7303a3e-e627-451c-9261-e553595fb53c-sys\") pod \"perf-node-gather-daemonset-bb5kk\" (UID: \"e7303a3e-e627-451c-9261-e553595fb53c\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"
Sep 07 23:09:04.429485 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.429366    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7303a3e-e627-451c-9261-e553595fb53c-lib-modules\") pod \"perf-node-gather-daemonset-bb5kk\" (UID: \"e7303a3e-e627-451c-9261-e553595fb53c\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"
Sep 07 23:09:04.429485 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.429390    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e7303a3e-e627-451c-9261-e553595fb53c-proc\") pod \"perf-node-gather-daemonset-bb5kk\" (UID: \"e7303a3e-e627-451c-9261-e553595fb53c\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"
Sep 07 23:09:04.429485 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.429395    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e7303a3e-e627-451c-9261-e553595fb53c-podres\") pod \"perf-node-gather-daemonset-bb5kk\" (UID: \"e7303a3e-e627-451c-9261-e553595fb53c\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"
Sep 07 23:09:04.429485 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.429469    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7303a3e-e627-451c-9261-e553595fb53c-sys\") pod \"perf-node-gather-daemonset-bb5kk\" (UID: \"e7303a3e-e627-451c-9261-e553595fb53c\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"
Sep 07 23:09:04.429693 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.429491    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e7303a3e-e627-451c-9261-e553595fb53c-podres\") pod \"perf-node-gather-daemonset-bb5kk\" (UID: \"e7303a3e-e627-451c-9261-e553595fb53c\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"
Sep 07 23:09:04.429693 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.429473    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trrzz\" (UniqueName: \"kubernetes.io/projected/e7303a3e-e627-451c-9261-e553595fb53c-kube-api-access-trrzz\") pod \"perf-node-gather-daemonset-bb5kk\" (UID: \"e7303a3e-e627-451c-9261-e553595fb53c\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"
Sep 07 23:09:04.429693 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.429532    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7303a3e-e627-451c-9261-e553595fb53c-lib-modules\") pod \"perf-node-gather-daemonset-bb5kk\" (UID: \"e7303a3e-e627-451c-9261-e553595fb53c\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"
Sep 07 23:09:04.436536 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.436464    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trrzz\" (UniqueName: \"kubernetes.io/projected/e7303a3e-e627-451c-9261-e553595fb53c-kube-api-access-trrzz\") pod \"perf-node-gather-daemonset-bb5kk\" (UID: \"e7303a3e-e627-451c-9261-e553595fb53c\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"
Sep 07 23:09:04.495674 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.495636    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"
Sep 07 23:09:04.767473 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.767444    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"]
Sep 07 23:09:04.770401 ip-10-0-138-99 kubenswrapper[2567]: W0907 23:09:04.770369    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode7303a3e_e627_451c_9261_e553595fb53c.slice/crio-2c9278059766aa3aa8f52c5a507796736993d058bb3eca8284e64ed8ff8fc328 WatchSource:0}: Error finding container 2c9278059766aa3aa8f52c5a507796736993d058bb3eca8284e64ed8ff8fc328: Status 404 returned error can't find the container with id 2c9278059766aa3aa8f52c5a507796736993d058bb3eca8284e64ed8ff8fc328
Sep 07 23:09:04.935124 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.935086    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk" event={"ID":"e7303a3e-e627-451c-9261-e553595fb53c","Type":"ContainerStarted","Data":"f2a3fa01704438d3dfbff9589ab2add763b2222bc654e2dff352d4bee1c8f9fd"}
Sep 07 23:09:04.935322 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.935138    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk" event={"ID":"e7303a3e-e627-451c-9261-e553595fb53c","Type":"ContainerStarted","Data":"2c9278059766aa3aa8f52c5a507796736993d058bb3eca8284e64ed8ff8fc328"}
Sep 07 23:09:04.935322 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.935213    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"
Sep 07 23:09:04.948104 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:04.948052    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk" podStartSLOduration=0.94803447 podStartE2EDuration="948.03447ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 23:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:09:04.947120072 +0000 UTC m=+2228.003953032" watchObservedRunningTime="2026-09-07 23:09:04.94803447 +0000 UTC m=+2228.004867430"
Sep 07 23:09:05.551624 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:05.551590    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cwg2q_90c9c42a-571f-4e9c-a741-c4ac016b1308/dns/0.log"
Sep 07 23:09:05.569190 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:05.569161    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cwg2q_90c9c42a-571f-4e9c-a741-c4ac016b1308/kube-rbac-proxy/0.log"
Sep 07 23:09:05.705983 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:05.705944    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vb75n_b09b22bc-14fb-423b-b7b4-0d3620da9bca/dns-node-resolver/0.log"
Sep 07 23:09:06.201699 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:06.201666    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cz944_56a27e39-3141-4765-8e18-d8195a6c567e/node-ca/0.log"
Sep 07 23:09:07.484721 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:07.484691    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68c66dfc89-vdgsz_f0f0ca3c-6829-439a-b38b-642d0b090c9a/kube-auth-proxy/0.log"
Sep 07 23:09:07.595817 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:07.595773    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-6vqtn_d66e33da-96fe-433d-9dd2-3e5f1ef9f021/payload-processing/0.log"
Sep 07 23:09:08.456117 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:08.456087    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lxqwk_a7e100c0-fa8b-4565-978b-f3fa81227e5a/serve-healthcheck-canary/0.log"
Sep 07 23:09:09.045350 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:09.045321    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pqgx4_db59dacc-3974-4a74-91ae-1b083c32f17a/kube-rbac-proxy/0.log"
Sep 07 23:09:09.064672 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:09.064641    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pqgx4_db59dacc-3974-4a74-91ae-1b083c32f17a/exporter/0.log"
Sep 07 23:09:09.082170 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:09.082138    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pqgx4_db59dacc-3974-4a74-91ae-1b083c32f17a/extractor/0.log"
Sep 07 23:09:10.949714 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:10.949683    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bb5kk"
Sep 07 23:09:11.312462 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:11.312368    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65f4bf68db-vfwnw_c757c7f9-da23-43f1-821a-7b95b863ae71/manager/0.log"
Sep 07 23:09:11.580997 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:11.580964    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d648d744d-s9jnn_aee02241-2234-4d9e-a6c5-3ae7b3ad8d13/manager/0.log"
Sep 07 23:09:11.595583 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:11.595554    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d648d744d-s9jnn_aee02241-2234-4d9e-a6c5-3ae7b3ad8d13/copy-manifests/0.log"
Sep 07 23:09:12.737902 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:12.737869    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-575c877bb5-zm9xn_40abc3f9-3165-4fe2-a06b-90483abc6ab2/manager/0.log"
Sep 07 23:09:14.561699 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:14.561667    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68c66dfc89-vdgsz_f0f0ca3c-6829-439a-b38b-642d0b090c9a/kube-auth-proxy/0.log"
Sep 07 23:09:14.657063 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:14.657027    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-6vqtn_d66e33da-96fe-433d-9dd2-3e5f1ef9f021/payload-processing/0.log"
Sep 07 23:09:16.791691 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:16.791593    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68c66dfc89-vdgsz_f0f0ca3c-6829-439a-b38b-642d0b090c9a/kube-auth-proxy/0.log"
Sep 07 23:09:16.896552 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:16.896493    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-6vqtn_d66e33da-96fe-433d-9dd2-3e5f1ef9f021/payload-processing/0.log"
Sep 07 23:09:19.694454 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:19.694380    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x7bn4_d9a478e0-d256-4ab9-9e76-6a751a9b34f7/kube-multus-additional-cni-plugins/0.log"
Sep 07 23:09:19.711467 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:19.711437    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x7bn4_d9a478e0-d256-4ab9-9e76-6a751a9b34f7/egress-router-binary-copy/0.log"
Sep 07 23:09:19.727800 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:19.727773    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x7bn4_d9a478e0-d256-4ab9-9e76-6a751a9b34f7/cni-plugins/0.log"
Sep 07 23:09:19.745753 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:19.745725    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x7bn4_d9a478e0-d256-4ab9-9e76-6a751a9b34f7/bond-cni-plugin/0.log"
Sep 07 23:09:19.764329 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:19.764294    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x7bn4_d9a478e0-d256-4ab9-9e76-6a751a9b34f7/routeoverride-cni/0.log"
Sep 07 23:09:19.781484 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:19.781443    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x7bn4_d9a478e0-d256-4ab9-9e76-6a751a9b34f7/whereabouts-cni-bincopy/0.log"
Sep 07 23:09:19.797528 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:19.797497    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x7bn4_d9a478e0-d256-4ab9-9e76-6a751a9b34f7/whereabouts-cni/0.log"
Sep 07 23:09:19.851961 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:19.851928    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-b8xx6_afca2585-5d1b-4577-8efc-882374f40887/kube-multus/0.log"
Sep 07 23:09:19.980159 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:19.980061    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wrtvx_f95816cb-c07f-4e9f-b828-930dcf1c06eb/network-metrics-daemon/0.log"
Sep 07 23:09:19.991979 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:19.991948    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wrtvx_f95816cb-c07f-4e9f-b828-930dcf1c06eb/kube-rbac-proxy/0.log"
Sep 07 23:09:20.762903 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:20.762865    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fnj7_0f2dfdc2-be64-43c6-813f-f34f09577b9b/ovn-controller/0.log"
Sep 07 23:09:20.775926 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:20.775896    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fnj7_0f2dfdc2-be64-43c6-813f-f34f09577b9b/ovn-acl-logging/0.log"
Sep 07 23:09:20.799741 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:20.799707    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fnj7_0f2dfdc2-be64-43c6-813f-f34f09577b9b/kube-rbac-proxy-node/0.log"
Sep 07 23:09:20.816163 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:20.816138    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fnj7_0f2dfdc2-be64-43c6-813f-f34f09577b9b/kube-rbac-proxy-ovn-metrics/0.log"
Sep 07 23:09:20.830839 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:20.830804    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fnj7_0f2dfdc2-be64-43c6-813f-f34f09577b9b/northd/0.log"
Sep 07 23:09:20.847053 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:20.847024    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fnj7_0f2dfdc2-be64-43c6-813f-f34f09577b9b/nbdb/0.log"
Sep 07 23:09:20.864017 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:20.863986    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fnj7_0f2dfdc2-be64-43c6-813f-f34f09577b9b/sbdb/0.log"
Sep 07 23:09:20.952854 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:20.952820    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fnj7_0f2dfdc2-be64-43c6-813f-f34f09577b9b/ovnkube-controller/0.log"
Sep 07 23:09:22.345816 ip-10-0-138-99 kubenswrapper[2567]: I0907 23:09:22.345786    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-q6zr5_0a7d1434-99e1-4ab9-8b17-08aebf32c93a/network-check-target-container/0.log"