Jul 24 17:40:59.450627 ip-10-0-137-93 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 24 17:40:59.450637 ip-10-0-137-93 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 24 17:40:59.450644 ip-10-0-137-93 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 24 17:40:59.450872 ip-10-0-137-93 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 24 17:41:09.492167 ip-10-0-137-93 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Jul 24 17:41:09.492182 ip-10-0-137-93 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6c33531efcd0480fbc2422182f56e6a5 --
Jul 24 17:42:56.199533 ip-10-0-137-93 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 17:42:56.567406 ip-10-0-137-93 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 17:42:56.567406 ip-10-0-137-93 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 17:42:56.567406 ip-10-0-137-93 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 17:42:56.567406 ip-10-0-137-93 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 17:42:56.567406 ip-10-0-137-93 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 17:42:56.569361 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.569247    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 17:42:56.572552 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572536    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:42:56.572591 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572554    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:42:56.572591 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572559    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:42:56.572591 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572562    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:42:56.572591 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572565    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:42:56.572591 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572569    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:42:56.572591 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572572    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:42:56.572591 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572575    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:42:56.572591 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572578    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:42:56.572591 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572581    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:42:56.572591 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572584    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:42:56.572591 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572586    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:42:56.572591 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572589    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:42:56.572591 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572592    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:42:56.572591 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572595    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:42:56.572591 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572598    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572601    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572604    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572606    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572609    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572612    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572614    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572617    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572620    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572623    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572625    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572628    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572630    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572633    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572635    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572638    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572641    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572643    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572646    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572649    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:42:56.572953 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572651    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572653    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572656    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572659    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572661    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572664    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572666    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572669    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572671    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572674    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572678    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572681    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572684    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572687    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572691    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572694    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572697    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572700    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572702    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572705    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:42:56.573487 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572707    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:42:56.573972 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572710    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:42:56.573972 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572713    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:42:56.573972 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572716    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:42:56.573972 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572718    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:42:56.573972 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572721    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:42:56.573972 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572723    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:42:56.573972 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572726    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:42:56.573972 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572728    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:42:56.573972 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572731    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:42:56.573972 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572733    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:42:56.573972 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572737    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:42:56.573972 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572739    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:42:56.573972 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572742    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:42:56.573972 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572745    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:42:56.573972 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572747    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:42:56.573972 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572751    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:42:56.573972 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572755    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:42:56.573972 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572758    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:42:56.573972 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572760    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572763    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572765    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572768    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572770    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572773    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572775    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572779    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572782    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572784    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572787    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572789    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572791    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572794    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572797    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572799    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572802    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572804    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572806    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572809    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:42:56.574441 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572811    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572814    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572817    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572819    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.572821    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573533    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573541    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573544    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573546    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573549    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573552    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573555    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573558    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573560    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573563    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573567    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573571    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573574    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573577    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573581    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:42:56.574926 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573584    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573587    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573590    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573592    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573595    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573597    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573600    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573603    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573605    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573608    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573611    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573613    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573616    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573619    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573621    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573624    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573627    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573629    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573632    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573634    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:42:56.575432 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573637    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:42:56.575922 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573639    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:42:56.575922 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573642    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:42:56.575922 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573646    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:42:56.575922 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573648    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:42:56.575922 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573651    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:42:56.575922 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573653    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:42:56.575922 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573655    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:42:56.575922 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573659    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:42:56.575922 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573661    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:42:56.575922 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573664    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:42:56.575922 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573667    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:42:56.575922 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573670    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:42:56.575922 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573672    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:42:56.575922 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573675    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:42:56.575922 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573679    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:42:56.575922 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573682    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:42:56.575922 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573685    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:42:56.575922 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573688    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:42:56.575922 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573690    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:42:56.576405 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573693    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:42:56.576405 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573695    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:42:56.576405 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573698    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:42:56.576405 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573700    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:42:56.576405 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573703    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:42:56.576405 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573707    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:42:56.576405 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573710    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:42:56.576405 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573714    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:42:56.576405 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573716    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:42:56.576405 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573719    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:42:56.576405 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573722    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:42:56.576405 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573725    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:42:56.576405 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573728    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:42:56.576405 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573731    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:42:56.576405 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573733    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:42:56.576405 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573736    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:42:56.576405 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573739    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:42:56.576405 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573742    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:42:56.576405 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573744    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573747    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573750    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573753    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573755    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573759    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573762    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573765    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573767    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573770    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573772    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573775    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573777    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573780    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573783    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573785    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573788    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573791    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573793    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573795    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:42:56.576907 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573798    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:42:56.577401 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573801    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:42:56.577401 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573803    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:42:56.577401 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573806    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:42:56.577401 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573809    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:42:56.577401 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.573811    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:42:56.577401 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574236    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:42:56.577401 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574243    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:42:56.577401 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574246    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:42:56.577401 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574249    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:42:56.577401 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574253    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:42:56.577401 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574257    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:42:56.577401 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574260    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:42:56.577401 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574262    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:42:56.577401 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574266    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:42:56.577401 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574270    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:42:56.577401 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574275    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:42:56.577401 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574279    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:42:56.577401 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574282    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:42:56.577401 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574286    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574288    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574291    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574294    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574297    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574299    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574302    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574305    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574307    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574310    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574314    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574317    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574320    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574323    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574326    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574328    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574331    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574334    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574336    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574339    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:42:56.577880 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574341    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574344    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574346    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574349    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574352    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574355    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574357    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574360    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574363    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574366    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574369    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574371    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574374    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574376    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574378    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574381    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574383    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574386    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574388    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574391    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:42:56.578399 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574393    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574396    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574399    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574401    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574404    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574406    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574409    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574411    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574413    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574416    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574418    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574421    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574423    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574425    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574428    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574430    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574433    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574436    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574439    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574441    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:42:56.578870 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574444    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:42:56.579418 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574447    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:42:56.579418 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574450    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:42:56.579418 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574453    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:42:56.579418 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574455    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:42:56.579418 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574458    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:42:56.579418 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574460    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:42:56.579418 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574463    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:42:56.579418 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574465    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:42:56.579418 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574468    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:42:56.579418 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574471    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:42:56.579418 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574473    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:42:56.579418 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574476    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:42:56.579418 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574478    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:42:56.579418 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574480    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:42:56.579418 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574483    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:42:56.579418 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574486    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:42:56.579418 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574488    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:42:56.579418 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574491    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:42:56.579418 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574493    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574496    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574499    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574501    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574504    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574507    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.574509    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574571    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574578    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574585    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574588    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574595    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574600    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574606    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574612    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574621    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574624    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574627    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574631    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574634    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574637    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574639    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574642    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 24 17:42:56.579902 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574645    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574648    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574653    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574656    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574659    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574662    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574665    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574669    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574672    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574675    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574679    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574681    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574684    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574687    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574691    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574694    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574698    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574701    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574704    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574707    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574711    2578 flags.go:64] FLAG: --enable-server="true"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574715    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574719    2578 flags.go:64] FLAG: --event-burst="100"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574722    2578 flags.go:64] FLAG: --event-qps="50"
Jul 24 17:42:56.580477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574727    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574730    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574733    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574736    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574740    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574743    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574746    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574748    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574751    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574754    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574757    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574760    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574763    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574766    2578 flags.go:64] FLAG: --feature-gates=""
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574769    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574772    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574775    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574778    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574781    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574784    2578 flags.go:64] FLAG: --help="false"
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574787    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-137-93.ec2.internal"
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574790    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574793    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 17:42:56.581044 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574796    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574799    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574803    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574806    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574808    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574812    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574815    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574819    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574822    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574826    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574829    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574832    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574835    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574837    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574840    2578 flags.go:64] FLAG: --lock-file=""
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574843    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574846    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574848    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574853    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574856    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574859    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574861    2578 flags.go:64] FLAG: --logging-format="text"
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574864    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574867    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 17:42:56.581699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574870    2578 flags.go:64] FLAG: --manifest-url=""
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574873    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574877    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574880    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574884    2578 flags.go:64] FLAG: --max-pods="110"
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574886    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574889    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574892    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574894    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574897    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574900    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574903    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574910    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574914    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574917    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574920    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574924    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574933    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574936    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574940    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574942    2578 flags.go:64] FLAG: --port="10250"
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574945    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574948    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0300c80a704a79186"
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574951    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 24 17:42:56.582323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574954    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574957    2578 flags.go:64] FLAG: --register-node="true"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574960    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574964    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574967    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574969    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574972    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574976    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574979    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574982    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574985    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574988    2578 flags.go:64] FLAG: --runonce="false"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574991    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574994    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574996    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.574999    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575002    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575005    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575008    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575011    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575013    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575016    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575020    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575023    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575026    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 17:42:56.582918 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575030    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575033    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575038    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575041    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575044    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575048    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575051    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575053    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575056    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575059    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575062    2578 flags.go:64] FLAG: --v="2"
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575065    2578 flags.go:64] FLAG: --version="false"
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575073    2578 flags.go:64] FLAG: --vmodule=""
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575077    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575080    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575193    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575197    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575200    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575203    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575206    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575209    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575223    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575226    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:42:56.583585 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575229    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575231    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575234    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575236    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575239    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575241    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575245    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575248    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575251    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575254    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575257    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575260    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575262    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575265    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575268    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575272    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575276    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575279    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575282    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575285    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:42:56.584200 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575287    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:42:56.584747 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575290    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:42:56.584747 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575293    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:42:56.584747 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575295    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:42:56.584747 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575297    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:42:56.584747 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575300    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:42:56.584747 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575303    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:42:56.584747 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575305    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:42:56.584747 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575309    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:42:56.584747 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575312    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:42:56.584747 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575314    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:42:56.584747 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575317    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:42:56.584747 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575319    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:42:56.584747 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575322    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:42:56.584747 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575324    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:42:56.584747 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575326    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:42:56.584747 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575334    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:42:56.584747 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575339    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:42:56.584747 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575343    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:42:56.584747 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575345    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575349    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575351    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575354    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575357    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575360    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575363    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575365    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575368    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575370    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575373    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575376    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575378    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575381    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575383    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575386    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575388    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575391    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575393    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575396    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:42:56.585271 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575398    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575401    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575404    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575406    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575409    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575411    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575414    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575416    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575419    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575421    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575424    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575427    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575430    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575433    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575436    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575438    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575441    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575444    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575447    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575449    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:42:56.585812 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575452    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:42:56.586325 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575455    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:42:56.586325 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575457    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:42:56.586325 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575460    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:42:56.586325 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575463    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:42:56.586325 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575465    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:42:56.586325 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575468    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:42:56.586325 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575470    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:42:56.586325 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575473    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:42:56.586325 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575475    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:42:56.586325 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575479    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:42:56.586325 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.575483    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:42:56.586325 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.575488    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 17:42:56.586325 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.582731    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 17:42:56.586325 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.582745    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 17:42:56.586689 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.582761    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 17:42:56.586689 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.582767    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 17:42:56.586689 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582822    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:42:56.586689 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582826    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:42:56.586689 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582830    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:42:56.586689 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582833    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:42:56.586689 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582836    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:42:56.586689 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582839    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:42:56.586689 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582842    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:42:56.586689 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582845    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:42:56.586689 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582848    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:42:56.586689 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582850    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:42:56.586689 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582853    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:42:56.586689 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582856    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:42:56.586689 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582860    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:42:56.586689 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582863    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:42:56.586689 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582866    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:42:56.586689 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582870    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:42:56.586689 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582873    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582876    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582879    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582882    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582885    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582887    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582890    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582893    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582895    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582898    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582900    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582903    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582906    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582909    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582912    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582915    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582918    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582920    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582923    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582926    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:42:56.587476 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582929    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582931    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582934    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582936    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582939    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582942    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582944    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582947    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582949    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582952    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582955    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582957    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582960    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582962    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582965    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582968    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582970    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582972    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582975    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582977    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:42:56.588005 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582980    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:42:56.588567 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582982    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:42:56.588567 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582985    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:42:56.588567 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582987    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:42:56.588567 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582990    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:42:56.588567 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582994    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:42:56.588567 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582996    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:42:56.588567 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.582999    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:42:56.588567 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583001    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:42:56.588567 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583004    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:42:56.588567 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583007    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:42:56.588567 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583009    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:42:56.588567 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583012    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:42:56.588567 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583014    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:42:56.588567 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583018    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:42:56.588567 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583022    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:42:56.588567 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583025    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:42:56.588567 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583028    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:42:56.588567 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583031    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:42:56.588567 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583034    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583037    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583039    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583042    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583045    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583048    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583051    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583053    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583056    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583058    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583061    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583063    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583065    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583068    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583070    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583073    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583076    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583078    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583082    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583084    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:42:56.589069 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583087    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:42:56.589758 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583090    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:42:56.589758 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583092    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:42:56.589758 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583095    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:42:56.589758 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.583100    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 17:42:56.589758 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583225    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:42:56.589758 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583231    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:42:56.589758 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583234    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:42:56.589758 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583237    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:42:56.589758 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583239    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:42:56.589758 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583242    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:42:56.589758 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583245    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:42:56.589758 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583247    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:42:56.589758 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583250    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:42:56.589758 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583252    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:42:56.589758 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583255    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:42:56.590312 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583258    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:42:56.590312 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583261    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:42:56.590312 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583263    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:42:56.590312 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583266    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:42:56.590312 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583268    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:42:56.590312 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583271    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:42:56.590312 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583273    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:42:56.590312 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583276    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:42:56.590312 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583278    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:42:56.590312 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583281    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:42:56.590312 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583283    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:42:56.590312 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583286    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:42:56.590312 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583288    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:42:56.590312 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583291    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:42:56.590312 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583294    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:42:56.590312 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583298    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:42:56.590312 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583301    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:42:56.590312 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583305    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:42:56.590312 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583307    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583310    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583313    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583316    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583319    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583322    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583325    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583327    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583336    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583341    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583344    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583346    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583349    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583351    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583354    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583358    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583360    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583363    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583365    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583368    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:42:56.590795 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583370    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:42:56.591310 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583373    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:42:56.591310 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583375    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:42:56.591310 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583377    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:42:56.591310 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583380    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:42:56.591310 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583382    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:42:56.591310 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583385    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:42:56.591310 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583388    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:42:56.591310 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583392    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:42:56.591310 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583395    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:42:56.591310 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583398    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:42:56.591310 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583400    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:42:56.591310 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583402    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:42:56.591310 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583405    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:42:56.591310 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583407    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:42:56.591310 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583410    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:42:56.591310 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583413    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:42:56.591310 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583417    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:42:56.591310 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583420    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:42:56.591310 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583423    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:42:56.591777 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583426    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:42:56.591777 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583428    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:42:56.591777 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583431    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:42:56.591777 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583433    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:42:56.591777 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583436    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:42:56.591777 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583438    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:42:56.591777 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583441    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:42:56.591777 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583443    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:42:56.591777 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583446    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:42:56.591777 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583449    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:42:56.591777 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583451    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:42:56.591777 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583454    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:42:56.591777 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583456    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:42:56.591777 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583458    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:42:56.591777 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583461    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:42:56.591777 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583463    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:42:56.591777 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583466    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:42:56.591777 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583468    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:42:56.591777 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583471    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:42:56.592297 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583474    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:42:56.592297 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583476    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:42:56.592297 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583479    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:42:56.592297 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583481    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:42:56.592297 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583484    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:42:56.592297 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583486    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:42:56.592297 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583489    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:42:56.592297 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583491    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:42:56.592297 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583494    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:42:56.592297 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583496    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:42:56.592297 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:56.583499    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:42:56.592297 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.583503    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 17:42:56.592297 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.583596    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 17:42:56.592297 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.585356    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 17:42:56.592297 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.586651    2578 server.go:1019] "Starting client certificate rotation"
Jul 24 17:42:56.592664 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.586759    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 17:42:56.592664 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.586808    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 17:42:56.592664 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.590991    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 17:42:56.595207 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.595192    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 17:42:56.604412 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.604397    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 24 17:42:56.606176 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.606162    2578 log.go:25] "Validated CRI v1 image API"
Jul 24 17:42:56.606968 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.606954    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 17:42:56.609780 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.609763    2578 fs.go:135] Filesystem UUIDs: map[25721309-0399-4c6b-bd37-00219a6b6b92:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 881a3917-bc80-4274-8140-6d53ddcccc89:/dev/nvme0n1p4]
Jul 24 17:42:56.609848 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.609780    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 17:42:56.615488 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.615379    2578 manager.go:216] Machine: {Timestamp:2026-07-24 17:42:56.613371454 +0000 UTC m=+0.304294952 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099811 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23670cd5329224adea5eec3f535f23 SystemUUID:ec23670c-d532-9224-adea-5eec3f535f23 BootID:6c33531e-fcd0-480f-bc24-22182f56e6a5 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fc:ea:ec:74:73 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fc:ea:ec:74:73 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:9e:0b:91:44:71 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 17:42:56.615488 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.615482    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 17:42:56.615612 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.615586    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 17:42:56.616147 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.616131    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 17:42:56.616968 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.616949    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 17:42:56.617110 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.616970    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-93.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 17:42:56.617674 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.617664    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 17:42:56.617709 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.617677    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 17:42:56.617709 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.617689    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 17:42:56.618316 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.618306    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 17:42:56.618968 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.618959    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 17:42:56.620422 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.620411    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 17:42:56.620527 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.620518    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 17:42:56.620563 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.620545    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 17:42:56.620563 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.620555    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 17:42:56.620621 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.620565    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 17:42:56.620621 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.620575    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 24 17:42:56.620621 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.620583    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 17:42:56.620979 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.620965    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 17:42:56.621031 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.620994    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 17:42:56.623173 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.623150    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 17:42:56.626706 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.626631    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 17:42:56.626802 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.626732    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 17:42:56.628314 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.628299    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 17:42:56.628382 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.628321    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 17:42:56.628382 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.628331    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 17:42:56.628382 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.628339    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 17:42:56.628382 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.628348    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 17:42:56.628382 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.628354    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 17:42:56.628382 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.628362    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 17:42:56.628382 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.628367    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 17:42:56.628382 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.628373    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 17:42:56.628382 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.628379    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 17:42:56.628382 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.628387    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 17:42:56.628652 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.628393    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 17:42:56.629194 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.629183    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 17:42:56.629249 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.629196    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 17:42:56.631924 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.631894    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-93.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 17:42:56.632415 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:56.632387    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 17:42:56.632454 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:56.632389    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-93.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 17:42:56.632653 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.632642    2578 server.go:1268] "Started kubelet"
Jul 24 17:42:56.632737 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.632714    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 17:42:56.633199 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.633166    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 17:42:56.633279 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.633238    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 17:42:56.633560 ip-10-0-137-93 systemd[1]: Started Kubernetes Kubelet.
Jul 24 17:42:56.634615 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.634491    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 17:42:56.635051 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.635032    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 17:42:56.638907 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:56.638887    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 17:42:56.639774 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:56.638828    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-93.ec2.internal.18c54ab6e07636e5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-93.ec2.internal,UID:ip-10-0-137-93.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-93.ec2.internal,},FirstTimestamp:2026-07-24 17:42:56.632616677 +0000 UTC m=+0.323540177,LastTimestamp:2026-07-24 17:42:56.632616677 +0000 UTC m=+0.323540177,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-93.ec2.internal,}"
Jul 24 17:42:56.640009 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.639997    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 17:42:56.640105 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.639999    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 17:42:56.640150 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:56.640091    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-93.ec2.internal\" not found"
Jul 24 17:42:56.640606 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.640487    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 17:42:56.640606 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.640503    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 17:42:56.640606 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.640557    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 17:42:56.640606 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.640569    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 17:42:56.640606 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.640591    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 17:42:56.640925 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.640906    2578 factory.go:55] Registering systemd factory
Jul 24 17:42:56.640925 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.640929    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 24 17:42:56.641197 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.641183    2578 factory.go:153] Registering CRI-O factory
Jul 24 17:42:56.641285 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.641200    2578 factory.go:223] Registration of the crio container factory successfully
Jul 24 17:42:56.641285 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.641262    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 17:42:56.641363 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.641289    2578 factory.go:103] Registering Raw factory
Jul 24 17:42:56.641363 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.641304    2578 manager.go:1196] Started watching for new ooms in manager
Jul 24 17:42:56.641738 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.641725    2578 manager.go:349] Starting recovery of all containers
Jul 24 17:42:56.644788 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:56.644760    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-93.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 24 17:42:56.644919 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:56.644897    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 24 17:42:56.652337 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.652308    2578 manager.go:354] Recovery completed
Jul 24 17:42:56.654891 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:56.654866    2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 24 17:42:56.658157 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.658144    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:42:56.660014 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.659995    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:42:56.660100 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.660028    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:42:56.660100 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.660042    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:42:56.660561 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.660548    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 17:42:56.660561 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.660559    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 17:42:56.660689 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.660569    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:42:56.660689 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.660573    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 17:42:56.661249 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.661199    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:42:56.661320 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.661259    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:42:56.661320 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.661269    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:42:56.661916 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:56.661857    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-93.ec2.internal.18c54ab6e2184279  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-93.ec2.internal,UID:ip-10-0-137-93.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-93.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-93.ec2.internal,},FirstTimestamp:2026-07-24 17:42:56.660013689 +0000 UTC m=+0.350937191,LastTimestamp:2026-07-24 17:42:56.660013689 +0000 UTC m=+0.350937191,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-93.ec2.internal,}"
Jul 24 17:42:56.662127 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.662116    2578 policy_none.go:49] "None policy: Start"
Jul 24 17:42:56.662157 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.662132    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 17:42:56.662157 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.662142    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 17:42:56.663114 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.663104    2578 policy_none.go:47] "Start"
Jul 24 17:42:56.672983 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:56.672908    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-93.ec2.internal.18c54ab6e21897da  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-93.ec2.internal,UID:ip-10-0-137-93.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-93.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-93.ec2.internal,},FirstTimestamp:2026-07-24 17:42:56.660035546 +0000 UTC m=+0.350959045,LastTimestamp:2026-07-24 17:42:56.660035546 +0000 UTC m=+0.350959045,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-93.ec2.internal,}"
Jul 24 17:42:56.684015 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:56.683848    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-93.ec2.internal.18c54ab6e218c059  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-93.ec2.internal,UID:ip-10-0-137-93.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-93.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-93.ec2.internal,},FirstTimestamp:2026-07-24 17:42:56.660045913 +0000 UTC m=+0.350969412,LastTimestamp:2026-07-24 17:42:56.660045913 +0000 UTC m=+0.350969412,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-93.ec2.internal,}"
Jul 24 17:42:56.692231 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:56.692149    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-93.ec2.internal.18c54ab6e2184279\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-93.ec2.internal.18c54ab6e2184279  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-93.ec2.internal,UID:ip-10-0-137-93.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-93.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-93.ec2.internal,},FirstTimestamp:2026-07-24 17:42:56.660013689 +0000 UTC m=+0.350937191,LastTimestamp:2026-07-24 17:42:56.661246647 +0000 UTC m=+0.352170146,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-93.ec2.internal,}"
Jul 24 17:42:56.696701 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.696684    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-95cs4"
Jul 24 17:42:56.697817 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.697806    2578 manager.go:337] "Starting Device Plugin manager"
Jul 24 17:42:56.697859 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:56.697834    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 17:42:56.697859 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.697845    2578 server.go:87] "Starting device plugin registration server"
Jul 24 17:42:56.698083 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.698073    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 17:42:56.698133 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.698084    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 17:42:56.698192 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.698178    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 17:42:56.698291 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.698269    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 17:42:56.698291 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.698281    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 17:42:56.698776 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:56.698751    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 17:42:56.698854 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:56.698807    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-93.ec2.internal\" not found"
Jul 24 17:42:56.700257 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:56.700186    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-93.ec2.internal.18c54ab6e21897da\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-93.ec2.internal.18c54ab6e21897da  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-93.ec2.internal,UID:ip-10-0-137-93.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-93.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-93.ec2.internal,},FirstTimestamp:2026-07-24 17:42:56.660035546 +0000 UTC m=+0.350959045,LastTimestamp:2026-07-24 17:42:56.661264056 +0000 UTC m=+0.352187555,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-93.ec2.internal,}"
Jul 24 17:42:56.704498 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.704478    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-95cs4"
Jul 24 17:42:56.792053 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.792019    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 17:42:56.793247 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.793230    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 17:42:56.793350 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.793268    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 17:42:56.793350 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.793290    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 17:42:56.793350 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:56.793322    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 17:42:56.795658 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.795637    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:42:56.798573 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.798557    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:42:56.799378 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.799364    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:42:56.799444 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.799391    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:42:56.799444 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.799402    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:42:56.799444 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.799423    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-93.ec2.internal"
Jul 24 17:42:56.805424 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:56.805410    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-93.ec2.internal"
Jul 24 17:42:56.805490 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:56.805429    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-93.ec2.internal\": node \"ip-10-0-137-93.ec2.internal\" not found"
Jul 24 17:42:56.818418 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:56.818342    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-93.ec2.internal\" not found"
Jul 24 17:42:58.061381 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.061346    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-93.ec2.internal"]
Jul 24 17:42:58.061502 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.061429    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:42:58.062320 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.062304    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:42:58.062377 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.062338    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:42:58.062377 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.062350    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:42:58.063427 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.063416    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:42:58.063533 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.063519    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal"
Jul 24 17:42:58.063571 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.063550    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:42:58.064150 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.064137    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:42:58.064212 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.064160    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:42:58.064212 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.064170    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:42:58.065073 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.065057    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:42:58.065119 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.065087    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:42:58.065119 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.065097    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:42:58.065177 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.065166    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-93.ec2.internal"
Jul 24 17:42:58.065207 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.065187    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:42:58.065825 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.065811    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:42:58.065895 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.065833    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:42:58.065895 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.065846    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:42:58.086336 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:58.086319    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-93.ec2.internal\" not found"
Jul 24 17:42:58.094390 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:58.094374    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-93.ec2.internal\" not found" node="ip-10-0-137-93.ec2.internal"
Jul 24 17:42:58.098749 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:58.098735    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-93.ec2.internal\" not found" node="ip-10-0-137-93.ec2.internal"
Jul 24 17:42:58.109662 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.109643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b0743ffe14dbbea13ef8e30cf5bd9ed2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal\" (UID: \"b0743ffe14dbbea13ef8e30cf5bd9ed2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal"
Jul 24 17:42:58.109733 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.109669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0743ffe14dbbea13ef8e30cf5bd9ed2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal\" (UID: \"b0743ffe14dbbea13ef8e30cf5bd9ed2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal"
Jul 24 17:42:58.109733 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.109687    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b13de3ecc1d0e8f58103000ebb6cab18-config\") pod \"kube-apiserver-proxy-ip-10-0-137-93.ec2.internal\" (UID: \"b13de3ecc1d0e8f58103000ebb6cab18\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-93.ec2.internal"
Jul 24 17:42:58.186525 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:58.186502    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-93.ec2.internal\" not found"
Jul 24 17:42:58.209869 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.209848    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b0743ffe14dbbea13ef8e30cf5bd9ed2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal\" (UID: \"b0743ffe14dbbea13ef8e30cf5bd9ed2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal"
Jul 24 17:42:58.209934 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.209874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0743ffe14dbbea13ef8e30cf5bd9ed2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal\" (UID: \"b0743ffe14dbbea13ef8e30cf5bd9ed2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal"
Jul 24 17:42:58.209934 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.209891    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b13de3ecc1d0e8f58103000ebb6cab18-config\") pod \"kube-apiserver-proxy-ip-10-0-137-93.ec2.internal\" (UID: \"b13de3ecc1d0e8f58103000ebb6cab18\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-93.ec2.internal"
Jul 24 17:42:58.209934 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.209919    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b13de3ecc1d0e8f58103000ebb6cab18-config\") pod \"kube-apiserver-proxy-ip-10-0-137-93.ec2.internal\" (UID: \"b13de3ecc1d0e8f58103000ebb6cab18\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-93.ec2.internal"
Jul 24 17:42:58.210026 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.209959    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b0743ffe14dbbea13ef8e30cf5bd9ed2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal\" (UID: \"b0743ffe14dbbea13ef8e30cf5bd9ed2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal"
Jul 24 17:42:58.210026 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.209991    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0743ffe14dbbea13ef8e30cf5bd9ed2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal\" (UID: \"b0743ffe14dbbea13ef8e30cf5bd9ed2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal"
Jul 24 17:42:58.287150 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:58.287083    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-93.ec2.internal\" not found"
Jul 24 17:42:58.387750 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:58.387712    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-93.ec2.internal\" not found"
Jul 24 17:42:58.396886 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.396865    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal"
Jul 24 17:42:58.401500 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.401485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-93.ec2.internal"
Jul 24 17:42:58.488411 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:58.488381    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-93.ec2.internal\" not found"
Jul 24 17:42:58.588898 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:58.588831    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-93.ec2.internal\" not found"
Jul 24 17:42:58.689413 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:58.689372    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-93.ec2.internal\" not found"
Jul 24 17:42:58.753831 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.753801    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 17:42:58.754236 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.753954    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 17:42:58.790184 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:58.790158    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-93.ec2.internal\" not found"
Jul 24 17:42:58.875981 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.875879    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 17:37:57 +0000 UTC" deadline="2028-02-08 11:29:05.02043201 +0000 UTC"
Jul 24 17:42:58.875981 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.875926    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13529h46m6.14450988s"
Jul 24 17:42:58.891040 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:58.891017    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-93.ec2.internal\" not found"
Jul 24 17:42:58.947445 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.947420    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:42:58.948593 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.948574    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:42:58.948697 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.948609    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:42:58.948697 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.948620    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:42:58.987919 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.987894    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:42:58.992857 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:58.992838    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:42:59.008711 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.008692    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal"
Jul 24 17:42:59.018827 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.018810    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 17:42:59.020902 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.020886    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-93.ec2.internal"
Jul 24 17:42:59.027160 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.027146    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 17:42:59.058059 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:59.058032    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0743ffe14dbbea13ef8e30cf5bd9ed2.slice/crio-ac44640f4f6db13decac2d9c27eee97d7c7ab27feed656e755e707846bde7b76 WatchSource:0}: Error finding container ac44640f4f6db13decac2d9c27eee97d7c7ab27feed656e755e707846bde7b76: Status 404 returned error can't find the container with id ac44640f4f6db13decac2d9c27eee97d7c7ab27feed656e755e707846bde7b76
Jul 24 17:42:59.058346 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:42:59.058322    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb13de3ecc1d0e8f58103000ebb6cab18.slice/crio-0111a4cade043184da86e0ee0780e4a3d4c06001559fb022c25a837e7b8945cf WatchSource:0}: Error finding container 0111a4cade043184da86e0ee0780e4a3d4c06001559fb022c25a837e7b8945cf: Status 404 returned error can't find the container with id 0111a4cade043184da86e0ee0780e4a3d4c06001559fb022c25a837e7b8945cf
Jul 24 17:42:59.062322 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.062308    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 17:42:59.138354 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.137448    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:42:59.790184 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.790155    2578 apiserver.go:52] "Watching apiserver"
Jul 24 17:42:59.800437 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.800410    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 17:42:59.801535 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.801486    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-jmdfh","openshift-image-registry/node-ca-bnjwn","openshift-network-diagnostics/network-check-target-8g49q","openshift-network-operator/iptables-alerter-vw29n","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf","openshift-dns/node-resolver-mn4d8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal","openshift-multus/multus-additional-cni-plugins-zsddc","openshift-multus/multus-jxx8w","openshift-multus/network-metrics-daemon-d4gbz","openshift-ovn-kubernetes/ovnkube-node-46bfx","kube-system/global-pull-secret-syncer-mmrr7","kube-system/konnectivity-agent-7b5js","kube-system/kube-apiserver-proxy-ip-10-0-137-93.ec2.internal"]
Jul 24 17:42:59.804263 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.804242    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.806763 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.806653    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.807485 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.807464    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pxh8b\""
Jul 24 17:42:59.807720 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.807704    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 17:42:59.807948 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.807932    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 17:42:59.808416 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.808130    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 17:42:59.808416 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.808240    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 17:42:59.808548 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.808508    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 17:42:59.809835 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.809688    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5mznr\""
Jul 24 17:42:59.809835 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.809729    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:42:59.809835 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:59.809809    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d4gbz" podUID="64e0f517-9969-47f1-8daf-8dfa194ba923"
Jul 24 17:42:59.810015 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.809856    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 17:42:59.810936 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.810673    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 17:42:59.811135 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.811119    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vw29n"
Jul 24 17:42:59.812735 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.812716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.813389 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.813371    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 17:42:59.814050 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.814028    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 17:42:59.814137 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.814094    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n5tzf\""
Jul 24 17:42:59.814137 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.814093    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 17:42:59.815101 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.815080    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 17:42:59.815195 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.815136    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mn4d8"
Jul 24 17:42:59.815254 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.815211    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 17:42:59.816307 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.815934    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 17:42:59.816307 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.815964    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jcfc6\""
Jul 24 17:42:59.817291 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.817273    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:42:59.817555 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:59.817347    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g49q" podUID="b4d50741-70f0-4e3b-90aa-f2fcaa98201f"
Jul 24 17:42:59.817723 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.817681    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v4s4s\""
Jul 24 17:42:59.817890 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.817870    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ffea99b-424f-4978-8cc0-be6971887c02-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.817982 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.817904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-etc-kubernetes\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.817982 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.817929    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ffea99b-424f-4978-8cc0-be6971887c02-system-cni-dir\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.817982 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.817954    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twp7j\" (UniqueName: \"kubernetes.io/projected/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-kube-api-access-twp7j\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.818149 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.817976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ffea99b-424f-4978-8cc0-be6971887c02-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.818149 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-cnibin\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.818149 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818044    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 17:42:59.818149 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-host-run-k8s-cni-cncf-io\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.818149 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-host-var-lib-kubelet\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.818149 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-multus-conf-dir\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.818149 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5cfde4d9-c9a9-4f63-adb4-84c845d4a76a-iptables-alerter-script\") pod \"iptables-alerter-vw29n\" (UID: \"5cfde4d9-c9a9-4f63-adb4-84c845d4a76a\") " pod="openshift-network-operator/iptables-alerter-vw29n"
Jul 24 17:42:59.818498 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818159    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.818498 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs\") pod \"network-metrics-daemon-d4gbz\" (UID: \"64e0f517-9969-47f1-8daf-8dfa194ba923\") " pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:42:59.818498 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818082    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 17:42:59.818498 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818214    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97d99\" (UniqueName: \"kubernetes.io/projected/64e0f517-9969-47f1-8daf-8dfa194ba923-kube-api-access-97d99\") pod \"network-metrics-daemon-d4gbz\" (UID: \"64e0f517-9969-47f1-8daf-8dfa194ba923\") " pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:42:59.818498 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ffea99b-424f-4978-8cc0-be6971887c02-cnibin\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.818498 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818262    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-system-cni-dir\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.818498 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ed65d95-64ef-4160-9849-30e2ae5e66d9-multus-daemon-config\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.818498 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818323    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9nqt\" (UniqueName: \"kubernetes.io/projected/9ed65d95-64ef-4160-9849-30e2ae5e66d9-kube-api-access-j9nqt\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.818498 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ffea99b-424f-4978-8cc0-be6971887c02-os-release\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.818498 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ffea99b-424f-4978-8cc0-be6971887c02-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.818498 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42ck9\" (UniqueName: \"kubernetes.io/projected/5ffea99b-424f-4978-8cc0-be6971887c02-kube-api-access-42ck9\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.818498 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ed65d95-64ef-4160-9849-30e2ae5e66d9-cni-binary-copy\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.818498 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-host-var-lib-cni-bin\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.818498 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-hostroot\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.819272 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818501    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-socket-dir\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.819272 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-registration-dir\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.819272 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-multus-cni-dir\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.819272 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-multus-socket-dir-parent\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.819272 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818633    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-host-run-netns\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.819272 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5cfde4d9-c9a9-4f63-adb4-84c845d4a76a-host-slash\") pod \"iptables-alerter-vw29n\" (UID: \"5cfde4d9-c9a9-4f63-adb4-84c845d4a76a\") " pod="openshift-network-operator/iptables-alerter-vw29n"
Jul 24 17:42:59.819272 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-sys-fs\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.819272 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818706    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ffea99b-424f-4978-8cc0-be6971887c02-cni-binary-copy\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.819272 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-os-release\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.819272 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xw77\" (UniqueName: \"kubernetes.io/projected/5cfde4d9-c9a9-4f63-adb4-84c845d4a76a-kube-api-access-6xw77\") pod \"iptables-alerter-vw29n\" (UID: \"5cfde4d9-c9a9-4f63-adb4-84c845d4a76a\") " pod="openshift-network-operator/iptables-alerter-vw29n"
Jul 24 17:42:59.819272 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818808    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-device-dir\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.819272 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-etc-selinux\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.819272 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-host-var-lib-cni-multus\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.819272 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.818876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-host-run-multus-certs\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.820598 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.820581    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bnjwn"
Jul 24 17:42:59.823473 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.823013    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 17:42:59.823473 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.823045    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:42:59.823473 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.823207    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8xnsg\""
Jul 24 17:42:59.823473 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.823361    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 17:42:59.823962 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.823864    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 17:42:59.824161 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.824112    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 17:42:59.825771 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.825730    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 17:42:59.825875 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.825852    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sdkc7\""
Jul 24 17:42:59.825931 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.825730    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 17:42:59.827013 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.826129    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.828002 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.827792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:42:59.828002 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:59.827851    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmrr7" podUID="4c849bf6-e357-4dca-8005-56fbbcb8198a"
Jul 24 17:42:59.828587 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.828568    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 17:42:59.829406 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.828878    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 17:42:59.829406 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.828966    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 17:42:59.829406 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.829091    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 17:42:59.829406 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.829205    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b4kjv\""
Jul 24 17:42:59.829406 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.829249    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 17:42:59.830754 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.830728    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7b5js"
Jul 24 17:42:59.831814 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.831344    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 17:42:59.835393 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.835263    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 17:42:59.835393 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.835370    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 17:42:59.835726 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.835602    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4bdwg\""
Jul 24 17:42:59.854617 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.854599    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ptqhz"
Jul 24 17:42:59.871862 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.871837    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ptqhz"
Jul 24 17:42:59.909433 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.909360    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 17:42:59.919164 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1d776dd-de07-4ba3-94f3-1b2492d09158-env-overrides\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.919304 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/afc9b6f4-a944-4228-97db-a0c9bbc70a7f-hosts-file\") pod \"node-resolver-mn4d8\" (UID: \"afc9b6f4-a944-4228-97db-a0c9bbc70a7f\") " pod="openshift-dns/node-resolver-mn4d8"
Jul 24 17:42:59.919304 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-host-var-lib-cni-multus\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.919304 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-slash\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.919304 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919275    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-host-var-lib-cni-multus\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.919507 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4c849bf6-e357-4dca-8005-56fbbcb8198a-kubelet-config\") pod \"global-pull-secret-syncer-mmrr7\" (UID: \"4c849bf6-e357-4dca-8005-56fbbcb8198a\") " pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:42:59.919507 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4c849bf6-e357-4dca-8005-56fbbcb8198a-dbus\") pod \"global-pull-secret-syncer-mmrr7\" (UID: \"4c849bf6-e357-4dca-8005-56fbbcb8198a\") " pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:42:59.919507 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-etc-kubernetes\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.919507 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-var-lib-kubelet\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:42:59.919507 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1d776dd-de07-4ba3-94f3-1b2492d09158-ovn-node-metrics-cert\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.919507 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret\") pod \"global-pull-secret-syncer-mmrr7\" (UID: \"4c849bf6-e357-4dca-8005-56fbbcb8198a\") " pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:42:59.919507 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919470    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-etc-kubernetes\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.919843 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twp7j\" (UniqueName: \"kubernetes.io/projected/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-kube-api-access-twp7j\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.919843 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-host-run-k8s-cni-cncf-io\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.919843 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-host-var-lib-kubelet\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.919843 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97d99\" (UniqueName: \"kubernetes.io/projected/64e0f517-9969-47f1-8daf-8dfa194ba923-kube-api-access-97d99\") pod \"network-metrics-daemon-d4gbz\" (UID: \"64e0f517-9969-47f1-8daf-8dfa194ba923\") " pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:42:59.919843 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccggd\" (UniqueName: \"kubernetes.io/projected/65c7cb53-638b-4934-9a59-ca69c44a4373-kube-api-access-ccggd\") pod \"node-ca-bnjwn\" (UID: \"65c7cb53-638b-4934-9a59-ca69c44a4373\") " pod="openshift-image-registry/node-ca-bnjwn"
Jul 24 17:42:59.919843 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-modprobe-d\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:42:59.919843 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-log-socket\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.919843 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919708    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5cfde4d9-c9a9-4f63-adb4-84c845d4a76a-iptables-alerter-script\") pod \"iptables-alerter-vw29n\" (UID: \"5cfde4d9-c9a9-4f63-adb4-84c845d4a76a\") " pod="openshift-network-operator/iptables-alerter-vw29n"
Jul 24 17:42:59.919843 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919733    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.919843 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919758    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-sysctl-conf\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:42:59.919843 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919795    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-tuned\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:42:59.919843 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-run-openvswitch\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.919843 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919842    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-cni-bin\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.920331 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.920331 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgwh7\" (UniqueName: \"kubernetes.io/projected/b1d776dd-de07-4ba3-94f3-1b2492d09158-kube-api-access-pgwh7\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.920331 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ffea99b-424f-4978-8cc0-be6971887c02-cnibin\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.920331 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-system-cni-dir\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.920331 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ed65d95-64ef-4160-9849-30e2ae5e66d9-multus-daemon-config\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.920331 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9nqt\" (UniqueName: \"kubernetes.io/projected/9ed65d95-64ef-4160-9849-30e2ae5e66d9-kube-api-access-j9nqt\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.920331 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.919985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-kubernetes\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:42:59.920331 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920000    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-var-lib-openvswitch\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.920331 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-cni-netd\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.920331 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920031    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-host-run-k8s-cni-cncf-io\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.920331 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42ck9\" (UniqueName: \"kubernetes.io/projected/5ffea99b-424f-4978-8cc0-be6971887c02-kube-api-access-42ck9\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.920331 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920057    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-host-var-lib-kubelet\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.920331 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-kubelet\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.920331 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-run-netns\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.920331 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1d776dd-de07-4ba3-94f3-1b2492d09158-ovnkube-script-lib\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.920331 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/789a3029-0741-46b0-969d-312e2181dbc8-agent-certs\") pod \"konnectivity-agent-7b5js\" (UID: \"789a3029-0741-46b0-969d-312e2181dbc8\") " pod="kube-system/konnectivity-agent-7b5js"
Jul 24 17:42:59.920331 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/789a3029-0741-46b0-969d-312e2181dbc8-konnectivity-ca\") pod \"konnectivity-agent-7b5js\" (UID: \"789a3029-0741-46b0-969d-312e2181dbc8\") " pod="kube-system/konnectivity-agent-7b5js"
Jul 24 17:42:59.921030 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxnc9\" (UniqueName: \"kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9\") pod \"network-check-target-8g49q\" (UID: \"b4d50741-70f0-4e3b-90aa-f2fcaa98201f\") " pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:42:59.921030 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-socket-dir\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.921030 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-registration-dir\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.921030 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920268    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1d776dd-de07-4ba3-94f3-1b2492d09158-ovnkube-config\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.921030 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920292    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.921030 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5cfde4d9-c9a9-4f63-adb4-84c845d4a76a-host-slash\") pod \"iptables-alerter-vw29n\" (UID: \"5cfde4d9-c9a9-4f63-adb4-84c845d4a76a\") " pod="openshift-network-operator/iptables-alerter-vw29n"
Jul 24 17:42:59.921030 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-sys-fs\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.921030 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920396    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5cfde4d9-c9a9-4f63-adb4-84c845d4a76a-host-slash\") pod \"iptables-alerter-vw29n\" (UID: \"5cfde4d9-c9a9-4f63-adb4-84c845d4a76a\") " pod="openshift-network-operator/iptables-alerter-vw29n"
Jul 24 17:42:59.921030 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-registration-dir\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.921030 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920433    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ffea99b-424f-4978-8cc0-be6971887c02-cnibin\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.921030 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-os-release\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.921030 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-host\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:42:59.921030 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xw77\" (UniqueName: \"kubernetes.io/projected/5cfde4d9-c9a9-4f63-adb4-84c845d4a76a-kube-api-access-6xw77\") pod \"iptables-alerter-vw29n\" (UID: \"5cfde4d9-c9a9-4f63-adb4-84c845d4a76a\") " pod="openshift-network-operator/iptables-alerter-vw29n"
Jul 24 17:42:59.921030 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920551    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-socket-dir\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.921030 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-device-dir\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.921030 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-etc-selinux\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.921030 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-host-run-multus-certs\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.921735 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65c7cb53-638b-4934-9a59-ca69c44a4373-host\") pod \"node-ca-bnjwn\" (UID: \"65c7cb53-638b-4934-9a59-ca69c44a4373\") " pod="openshift-image-registry/node-ca-bnjwn"
Jul 24 17:42:59.921735 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920687    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x7sq\" (UniqueName: \"kubernetes.io/projected/518b63cc-9a60-48bd-9961-2ee33a833b40-kube-api-access-6x7sq\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:42:59.921735 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-systemd-units\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.921735 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920737    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-etc-openvswitch\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.921735 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ffea99b-424f-4978-8cc0-be6971887c02-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.921735 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920796    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-system-cni-dir\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.921735 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/518b63cc-9a60-48bd-9961-2ee33a833b40-tmp\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:42:59.921735 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-lib-modules\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:42:59.921735 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920881    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5cfde4d9-c9a9-4f63-adb4-84c845d4a76a-iptables-alerter-script\") pod \"iptables-alerter-vw29n\" (UID: \"5cfde4d9-c9a9-4f63-adb4-84c845d4a76a\") " pod="openshift-network-operator/iptables-alerter-vw29n"
Jul 24 17:42:59.921735 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920887    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-host-run-multus-certs\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.921735 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ffea99b-424f-4978-8cc0-be6971887c02-system-cni-dir\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.921735 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ffea99b-424f-4978-8cc0-be6971887c02-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.921735 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.920990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-cnibin\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.921735 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-multus-conf-dir\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.921735 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs\") pod \"network-metrics-daemon-d4gbz\" (UID: \"64e0f517-9969-47f1-8daf-8dfa194ba923\") " pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:42:59.921735 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921067    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-sysctl-d\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:42:59.921735 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4blq9\" (UniqueName: \"kubernetes.io/projected/afc9b6f4-a944-4228-97db-a0c9bbc70a7f-kube-api-access-4blq9\") pod \"node-resolver-mn4d8\" (UID: \"afc9b6f4-a944-4228-97db-a0c9bbc70a7f\") " pod="openshift-dns/node-resolver-mn4d8"
Jul 24 17:42:59.922555 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-systemd\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:42:59.922555 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921206    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-run-ovn\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.922555 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-node-log\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.922555 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921275    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-run-ovn-kubernetes\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.922555 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ffea99b-424f-4978-8cc0-be6971887c02-os-release\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.922555 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ffea99b-424f-4978-8cc0-be6971887c02-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.922555 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ed65d95-64ef-4160-9849-30e2ae5e66d9-cni-binary-copy\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.922555 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ffea99b-424f-4978-8cc0-be6971887c02-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.922555 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ed65d95-64ef-4160-9849-30e2ae5e66d9-multus-daemon-config\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.922555 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-host-var-lib-cni-bin\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.922555 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921439    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-hostroot\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.922555 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921462    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-run\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:42:59.922555 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921476    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-sys-fs\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.922555 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-multus-cni-dir\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.922555 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-multus-socket-dir-parent\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.922555 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-device-dir\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.922555 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921529    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ffea99b-424f-4978-8cc0-be6971887c02-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.923340 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921607    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ffea99b-424f-4978-8cc0-be6971887c02-os-release\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.923340 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921684    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-multus-socket-dir-parent\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.923340 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921702    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-etc-selinux\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.923340 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921743    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-host-run-netns\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.923340 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-sysconfig\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:42:59.923340 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-run-systemd\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:42:59.923340 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afc9b6f4-a944-4228-97db-a0c9bbc70a7f-tmp-dir\") pod \"node-resolver-mn4d8\" (UID: \"afc9b6f4-a944-4228-97db-a0c9bbc70a7f\") " pod="openshift-dns/node-resolver-mn4d8"
Jul 24 17:42:59.923340 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ffea99b-424f-4978-8cc0-be6971887c02-cni-binary-copy\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.923340 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ffea99b-424f-4978-8cc0-be6971887c02-system-cni-dir\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.923340 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921938    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/65c7cb53-638b-4934-9a59-ca69c44a4373-serviceca\") pod \"node-ca-bnjwn\" (UID: \"65c7cb53-638b-4934-9a59-ca69c44a4373\") " pod="openshift-image-registry/node-ca-bnjwn"
Jul 24 17:42:59.923340 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921964    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-sys\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:42:59.923340 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-hostroot\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.923340 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.921983    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ffea99b-424f-4978-8cc0-be6971887c02-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.923340 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.922058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-host-run-netns\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.923340 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.922081    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-os-release\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.923340 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.922188    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-multus-cni-dir\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.923340 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.922250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-multus-conf-dir\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.923340 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.922293    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-cnibin\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.924057 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:59.922367    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:42:59.924057 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:42:59.922437    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs podName:64e0f517-9969-47f1-8daf-8dfa194ba923 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:00.422416664 +0000 UTC m=+2.945775146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs") pod "network-metrics-daemon-d4gbz" (UID: "64e0f517-9969-47f1-8daf-8dfa194ba923") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:42:59.924057 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.922498    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ffea99b-424f-4978-8cc0-be6971887c02-cni-binary-copy\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.924057 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.922525    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ed65d95-64ef-4160-9849-30e2ae5e66d9-cni-binary-copy\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.924057 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.922553    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ed65d95-64ef-4160-9849-30e2ae5e66d9-host-var-lib-cni-bin\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.928242 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.928182    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 17:42:59.932069 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.931975    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twp7j\" (UniqueName: \"kubernetes.io/projected/29579c17-0f23-4f2e-b0c8-7bb62dbecaec-kube-api-access-twp7j\") pod \"aws-ebs-csi-driver-node-8w2kf\" (UID: \"29579c17-0f23-4f2e-b0c8-7bb62dbecaec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:42:59.932069 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.931975    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9nqt\" (UniqueName: \"kubernetes.io/projected/9ed65d95-64ef-4160-9849-30e2ae5e66d9-kube-api-access-j9nqt\") pod \"multus-jxx8w\" (UID: \"9ed65d95-64ef-4160-9849-30e2ae5e66d9\") " pod="openshift-multus/multus-jxx8w"
Jul 24 17:42:59.932258 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.932234    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xw77\" (UniqueName: \"kubernetes.io/projected/5cfde4d9-c9a9-4f63-adb4-84c845d4a76a-kube-api-access-6xw77\") pod \"iptables-alerter-vw29n\" (UID: \"5cfde4d9-c9a9-4f63-adb4-84c845d4a76a\") " pod="openshift-network-operator/iptables-alerter-vw29n"
Jul 24 17:42:59.932645 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.932623    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42ck9\" (UniqueName: \"kubernetes.io/projected/5ffea99b-424f-4978-8cc0-be6971887c02-kube-api-access-42ck9\") pod \"multus-additional-cni-plugins-zsddc\" (UID: \"5ffea99b-424f-4978-8cc0-be6971887c02\") " pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:42:59.933510 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.933472    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97d99\" (UniqueName: \"kubernetes.io/projected/64e0f517-9969-47f1-8daf-8dfa194ba923-kube-api-access-97d99\") pod \"network-metrics-daemon-d4gbz\" (UID: \"64e0f517-9969-47f1-8daf-8dfa194ba923\") " pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:42:59.965972 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.965921    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal" event={"ID":"b0743ffe14dbbea13ef8e30cf5bd9ed2","Type":"ContainerStarted","Data":"ac44640f4f6db13decac2d9c27eee97d7c7ab27feed656e755e707846bde7b76"}
Jul 24 17:42:59.967393 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:42:59.967364    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-93.ec2.internal" event={"ID":"b13de3ecc1d0e8f58103000ebb6cab18","Type":"ContainerStarted","Data":"0111a4cade043184da86e0ee0780e4a3d4c06001559fb022c25a837e7b8945cf"}
Jul 24 17:43:00.022937 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.022892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4c849bf6-e357-4dca-8005-56fbbcb8198a-kubelet-config\") pod \"global-pull-secret-syncer-mmrr7\" (UID: \"4c849bf6-e357-4dca-8005-56fbbcb8198a\") " pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:00.022937 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.022938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4c849bf6-e357-4dca-8005-56fbbcb8198a-dbus\") pod \"global-pull-secret-syncer-mmrr7\" (UID: \"4c849bf6-e357-4dca-8005-56fbbcb8198a\") " pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:00.023176 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.022990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-var-lib-kubelet\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.023176 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1d776dd-de07-4ba3-94f3-1b2492d09158-ovn-node-metrics-cert\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.023176 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4c849bf6-e357-4dca-8005-56fbbcb8198a-dbus\") pod \"global-pull-secret-syncer-mmrr7\" (UID: \"4c849bf6-e357-4dca-8005-56fbbcb8198a\") " pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:00.023176 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret\") pod \"global-pull-secret-syncer-mmrr7\" (UID: \"4c849bf6-e357-4dca-8005-56fbbcb8198a\") " pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:00.023176 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccggd\" (UniqueName: \"kubernetes.io/projected/65c7cb53-638b-4934-9a59-ca69c44a4373-kube-api-access-ccggd\") pod \"node-ca-bnjwn\" (UID: \"65c7cb53-638b-4934-9a59-ca69c44a4373\") " pod="openshift-image-registry/node-ca-bnjwn"
Jul 24 17:43:00.023176 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-modprobe-d\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.023176 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-log-socket\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.023176 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-sysctl-conf\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.023547 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-tuned\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.023547 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-run-openvswitch\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.023547 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-cni-bin\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.023547 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.023547 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023295    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgwh7\" (UniqueName: \"kubernetes.io/projected/b1d776dd-de07-4ba3-94f3-1b2492d09158-kube-api-access-pgwh7\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.023547 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-kubernetes\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.023547 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-var-lib-openvswitch\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.023547 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-cni-netd\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.023547 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-kubelet\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.023547 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-run-netns\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.023547 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1d776dd-de07-4ba3-94f3-1b2492d09158-ovnkube-script-lib\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.023547 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/789a3029-0741-46b0-969d-312e2181dbc8-agent-certs\") pod \"konnectivity-agent-7b5js\" (UID: \"789a3029-0741-46b0-969d-312e2181dbc8\") " pod="kube-system/konnectivity-agent-7b5js"
Jul 24 17:43:00.023547 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/789a3029-0741-46b0-969d-312e2181dbc8-konnectivity-ca\") pod \"konnectivity-agent-7b5js\" (UID: \"789a3029-0741-46b0-969d-312e2181dbc8\") " pod="kube-system/konnectivity-agent-7b5js"
Jul 24 17:43:00.023547 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxnc9\" (UniqueName: \"kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9\") pod \"network-check-target-8g49q\" (UID: \"b4d50741-70f0-4e3b-90aa-f2fcaa98201f\") " pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:00.023547 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:00.023524    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:43:00.023547 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023538    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-var-lib-openvswitch\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.024330 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1d776dd-de07-4ba3-94f3-1b2492d09158-ovnkube-config\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.024330 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:00.023593    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret podName:4c849bf6-e357-4dca-8005-56fbbcb8198a nodeName:}" failed. No retries permitted until 2026-07-24 17:43:00.523571186 +0000 UTC m=+3.046929657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret") pod "global-pull-secret-syncer-mmrr7" (UID: "4c849bf6-e357-4dca-8005-56fbbcb8198a") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:43:00.024330 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023632    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.024330 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023640    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-cni-netd\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.024330 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023689    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-kubelet\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.024330 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-host\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.024330 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65c7cb53-638b-4934-9a59-ca69c44a4373-host\") pod \"node-ca-bnjwn\" (UID: \"65c7cb53-638b-4934-9a59-ca69c44a4373\") " pod="openshift-image-registry/node-ca-bnjwn"
Jul 24 17:43:00.024330 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023752    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-host\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.024330 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x7sq\" (UniqueName: \"kubernetes.io/projected/518b63cc-9a60-48bd-9961-2ee33a833b40-kube-api-access-6x7sq\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.024330 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-systemd-units\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.024330 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-etc-openvswitch\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.024330 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.024318    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/518b63cc-9a60-48bd-9961-2ee33a833b40-tmp\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.024877 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.024360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-lib-modules\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.024877 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.024415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-sysctl-d\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.024877 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.024472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4blq9\" (UniqueName: \"kubernetes.io/projected/afc9b6f4-a944-4228-97db-a0c9bbc70a7f-kube-api-access-4blq9\") pod \"node-resolver-mn4d8\" (UID: \"afc9b6f4-a944-4228-97db-a0c9bbc70a7f\") " pod="openshift-dns/node-resolver-mn4d8"
Jul 24 17:43:00.024877 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.024506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-systemd\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.026102 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.025394    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-kubernetes\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.026102 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.025736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1d776dd-de07-4ba3-94f3-1b2492d09158-ovnkube-config\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.026336 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.026313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1d776dd-de07-4ba3-94f3-1b2492d09158-ovn-node-metrics-cert\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.026451 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.024538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-run-ovn\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.026451 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.026410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-node-log\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.026451 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.026443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-run-ovn-kubernetes\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.026575 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.026479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-run\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.026575 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.026507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-sysconfig\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.026575 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.026534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-run-systemd\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.026575 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.026564    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afc9b6f4-a944-4228-97db-a0c9bbc70a7f-tmp-dir\") pod \"node-resolver-mn4d8\" (UID: \"afc9b6f4-a944-4228-97db-a0c9bbc70a7f\") " pod="openshift-dns/node-resolver-mn4d8"
Jul 24 17:43:00.026715 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.026595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/65c7cb53-638b-4934-9a59-ca69c44a4373-serviceca\") pod \"node-ca-bnjwn\" (UID: \"65c7cb53-638b-4934-9a59-ca69c44a4373\") " pod="openshift-image-registry/node-ca-bnjwn"
Jul 24 17:43:00.026715 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.026622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-sys\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.026715 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.026645    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1d776dd-de07-4ba3-94f3-1b2492d09158-env-overrides\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.026715 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.026673    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/afc9b6f4-a944-4228-97db-a0c9bbc70a7f-hosts-file\") pod \"node-resolver-mn4d8\" (UID: \"afc9b6f4-a944-4228-97db-a0c9bbc70a7f\") " pod="openshift-dns/node-resolver-mn4d8"
Jul 24 17:43:00.026715 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.026703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-slash\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.026943 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.026816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-slash\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.026943 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.026864    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-node-log\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.026943 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.026911    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-run-ovn-kubernetes\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.027036 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.026958    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-run\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.027036 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.027012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-sysconfig\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.027106 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.027053    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-run-systemd\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.027348 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.027330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afc9b6f4-a944-4228-97db-a0c9bbc70a7f-tmp-dir\") pod \"node-resolver-mn4d8\" (UID: \"afc9b6f4-a944-4228-97db-a0c9bbc70a7f\") " pod="openshift-dns/node-resolver-mn4d8"
Jul 24 17:43:00.027729 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.027709    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-run-netns\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.027819 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.027796    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-run-openvswitch\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.028311 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.028290    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1d776dd-de07-4ba3-94f3-1b2492d09158-ovnkube-script-lib\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.028733 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.028713    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/65c7cb53-638b-4934-9a59-ca69c44a4373-serviceca\") pod \"node-ca-bnjwn\" (UID: \"65c7cb53-638b-4934-9a59-ca69c44a4373\") " pod="openshift-image-registry/node-ca-bnjwn"
Jul 24 17:43:00.028829 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.028803    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-sys\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.029175 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.029155    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1d776dd-de07-4ba3-94f3-1b2492d09158-env-overrides\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.029251 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.029235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/afc9b6f4-a944-4228-97db-a0c9bbc70a7f-hosts-file\") pod \"node-resolver-mn4d8\" (UID: \"afc9b6f4-a944-4228-97db-a0c9bbc70a7f\") " pod="openshift-dns/node-resolver-mn4d8"
Jul 24 17:43:00.029343 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.029324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-systemd-units\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.029402 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.029323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-var-lib-kubelet\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.029516 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.029492    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65c7cb53-638b-4934-9a59-ca69c44a4373-host\") pod \"node-ca-bnjwn\" (UID: \"65c7cb53-638b-4934-9a59-ca69c44a4373\") " pod="openshift-image-registry/node-ca-bnjwn"
Jul 24 17:43:00.029690 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.029673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-sysctl-conf\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.029752 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023096    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4c849bf6-e357-4dca-8005-56fbbcb8198a-kubelet-config\") pod \"global-pull-secret-syncer-mmrr7\" (UID: \"4c849bf6-e357-4dca-8005-56fbbcb8198a\") " pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:00.029823 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.029809    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/789a3029-0741-46b0-969d-312e2181dbc8-konnectivity-ca\") pod \"konnectivity-agent-7b5js\" (UID: \"789a3029-0741-46b0-969d-312e2181dbc8\") " pod="kube-system/konnectivity-agent-7b5js"
Jul 24 17:43:00.029877 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.029821    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-modprobe-d\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.029877 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.029872    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-log-socket\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.030042 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.023592    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-host-cni-bin\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.030196 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.030176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-sysctl-d\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.030276 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.030257    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-lib-modules\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.030355 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.030338    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-systemd\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.030421 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.030397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-etc-openvswitch\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.031136 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.030523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1d776dd-de07-4ba3-94f3-1b2492d09158-run-ovn\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.031338 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.031320    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/789a3029-0741-46b0-969d-312e2181dbc8-agent-certs\") pod \"konnectivity-agent-7b5js\" (UID: \"789a3029-0741-46b0-969d-312e2181dbc8\") " pod="kube-system/konnectivity-agent-7b5js"
Jul 24 17:43:00.032746 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.032723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/518b63cc-9a60-48bd-9961-2ee33a833b40-etc-tuned\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.033621 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.033577    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/518b63cc-9a60-48bd-9961-2ee33a833b40-tmp\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.037198 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:00.037169    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:43:00.037198 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:00.037197    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:43:00.037349 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:00.037211    2578 projected.go:196] Error preparing data for projected volume kube-api-access-vxnc9 for pod openshift-network-diagnostics/network-check-target-8g49q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:43:00.037349 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:00.037279    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9 podName:b4d50741-70f0-4e3b-90aa-f2fcaa98201f nodeName:}" failed. No retries permitted until 2026-07-24 17:43:00.537260663 +0000 UTC m=+3.060619137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vxnc9" (UniqueName: "kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9") pod "network-check-target-8g49q" (UID: "b4d50741-70f0-4e3b-90aa-f2fcaa98201f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:43:00.038129 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.038105    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgwh7\" (UniqueName: \"kubernetes.io/projected/b1d776dd-de07-4ba3-94f3-1b2492d09158-kube-api-access-pgwh7\") pod \"ovnkube-node-46bfx\" (UID: \"b1d776dd-de07-4ba3-94f3-1b2492d09158\") " pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.038449 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.038429    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x7sq\" (UniqueName: \"kubernetes.io/projected/518b63cc-9a60-48bd-9961-2ee33a833b40-kube-api-access-6x7sq\") pod \"tuned-jmdfh\" (UID: \"518b63cc-9a60-48bd-9961-2ee33a833b40\") " pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.039481 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.039458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccggd\" (UniqueName: \"kubernetes.io/projected/65c7cb53-638b-4934-9a59-ca69c44a4373-kube-api-access-ccggd\") pod \"node-ca-bnjwn\" (UID: \"65c7cb53-638b-4934-9a59-ca69c44a4373\") " pod="openshift-image-registry/node-ca-bnjwn"
Jul 24 17:43:00.039759 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.039734    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4blq9\" (UniqueName: \"kubernetes.io/projected/afc9b6f4-a944-4228-97db-a0c9bbc70a7f-kube-api-access-4blq9\") pod \"node-resolver-mn4d8\" (UID: \"afc9b6f4-a944-4228-97db-a0c9bbc70a7f\") " pod="openshift-dns/node-resolver-mn4d8"
Jul 24 17:43:00.120283 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.120208    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zsddc"
Jul 24 17:43:00.128292 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.128265    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jxx8w"
Jul 24 17:43:00.142004 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.141986    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vw29n"
Jul 24 17:43:00.146795 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.146765    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf"
Jul 24 17:43:00.152437 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.152421    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mn4d8"
Jul 24 17:43:00.159893 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.159878    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bnjwn"
Jul 24 17:43:00.167500 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.167481    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jmdfh"
Jul 24 17:43:00.175001 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.174984    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:00.179491 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.179470    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:43:00.180525 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.180507    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7b5js"
Jul 24 17:43:00.429181 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.429108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs\") pod \"network-metrics-daemon-d4gbz\" (UID: \"64e0f517-9969-47f1-8daf-8dfa194ba923\") " pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:00.429323 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:00.429258    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:43:00.429323 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:00.429315    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs podName:64e0f517-9969-47f1-8daf-8dfa194ba923 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:01.429300328 +0000 UTC m=+3.952658797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs") pod "network-metrics-daemon-d4gbz" (UID: "64e0f517-9969-47f1-8daf-8dfa194ba923") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:43:00.529494 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.529453    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret\") pod \"global-pull-secret-syncer-mmrr7\" (UID: \"4c849bf6-e357-4dca-8005-56fbbcb8198a\") " pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:00.529693 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:00.529633    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:43:00.529762 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:00.529712    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret podName:4c849bf6-e357-4dca-8005-56fbbcb8198a nodeName:}" failed. No retries permitted until 2026-07-24 17:43:01.529690153 +0000 UTC m=+4.053048613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret") pod "global-pull-secret-syncer-mmrr7" (UID: "4c849bf6-e357-4dca-8005-56fbbcb8198a") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:43:00.630601 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.630567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxnc9\" (UniqueName: \"kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9\") pod \"network-check-target-8g49q\" (UID: \"b4d50741-70f0-4e3b-90aa-f2fcaa98201f\") " pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:00.630774 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:00.630757    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:43:00.630845 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:00.630796    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:43:00.630845 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:00.630811    2578 projected.go:196] Error preparing data for projected volume kube-api-access-vxnc9 for pod openshift-network-diagnostics/network-check-target-8g49q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:43:00.630940 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:00.630872    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9 podName:b4d50741-70f0-4e3b-90aa-f2fcaa98201f nodeName:}" failed. No retries permitted until 2026-07-24 17:43:01.63085305 +0000 UTC m=+4.154211526 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vxnc9" (UniqueName: "kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9") pod "network-check-target-8g49q" (UID: "b4d50741-70f0-4e3b-90aa-f2fcaa98201f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:43:00.875725 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.875695    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 17:37:59 +0000 UTC" deadline="2027-12-25 19:59:36.984102275 +0000 UTC"
Jul 24 17:43:00.875725 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.875722    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12458h16m36.108382768s"
Jul 24 17:43:00.885084 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:43:00.884911    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed65d95_64ef_4160_9849_30e2ae5e66d9.slice/crio-ed383cd30f0ed2d9d3db947132ce2953c8f71a35addc58ae94ff2be6eec36242 WatchSource:0}: Error finding container ed383cd30f0ed2d9d3db947132ce2953c8f71a35addc58ae94ff2be6eec36242: Status 404 returned error can't find the container with id ed383cd30f0ed2d9d3db947132ce2953c8f71a35addc58ae94ff2be6eec36242
Jul 24 17:43:00.886593 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:43:00.886496    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ffea99b_424f_4978_8cc0_be6971887c02.slice/crio-d72ec94c24576c0001871f04762f2d97a65116fb40642cc053088c7695cdebe6 WatchSource:0}: Error finding container d72ec94c24576c0001871f04762f2d97a65116fb40642cc053088c7695cdebe6: Status 404 returned error can't find the container with id d72ec94c24576c0001871f04762f2d97a65116fb40642cc053088c7695cdebe6
Jul 24 17:43:00.889252 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:43:00.889225    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29579c17_0f23_4f2e_b0c8_7bb62dbecaec.slice/crio-bd41d0885bf8f838dce3417818865d1838b88e4ccd10d0b42f8f9a7b373367c9 WatchSource:0}: Error finding container bd41d0885bf8f838dce3417818865d1838b88e4ccd10d0b42f8f9a7b373367c9: Status 404 returned error can't find the container with id bd41d0885bf8f838dce3417818865d1838b88e4ccd10d0b42f8f9a7b373367c9
Jul 24 17:43:00.890088 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:43:00.890062    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518b63cc_9a60_48bd_9961_2ee33a833b40.slice/crio-55e710feb205993703d1ce8a34b254531ebee09181fcb0a51fa835f1a3cbaf89 WatchSource:0}: Error finding container 55e710feb205993703d1ce8a34b254531ebee09181fcb0a51fa835f1a3cbaf89: Status 404 returned error can't find the container with id 55e710feb205993703d1ce8a34b254531ebee09181fcb0a51fa835f1a3cbaf89
Jul 24 17:43:00.891058 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:43:00.891028    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc9b6f4_a944_4228_97db_a0c9bbc70a7f.slice/crio-4de638b948ef9af7eda262ebede9db50a8963b33cf3096d8afa3d189c71fd241 WatchSource:0}: Error finding container 4de638b948ef9af7eda262ebede9db50a8963b33cf3096d8afa3d189c71fd241: Status 404 returned error can't find the container with id 4de638b948ef9af7eda262ebede9db50a8963b33cf3096d8afa3d189c71fd241
Jul 24 17:43:00.891883 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:43:00.891735    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d776dd_de07_4ba3_94f3_1b2492d09158.slice/crio-a9debe73fda1dcf57b22acd671c9412075b8e72e6c0b866da0ccf4fe2a8085ef WatchSource:0}: Error finding container a9debe73fda1dcf57b22acd671c9412075b8e72e6c0b866da0ccf4fe2a8085ef: Status 404 returned error can't find the container with id a9debe73fda1dcf57b22acd671c9412075b8e72e6c0b866da0ccf4fe2a8085ef
Jul 24 17:43:00.893080 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:43:00.892728    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789a3029_0741_46b0_969d_312e2181dbc8.slice/crio-d05be9e52168da50a0432838a707ed5d6f5057d41d4dd55e98c9ef92c75eb629 WatchSource:0}: Error finding container d05be9e52168da50a0432838a707ed5d6f5057d41d4dd55e98c9ef92c75eb629: Status 404 returned error can't find the container with id d05be9e52168da50a0432838a707ed5d6f5057d41d4dd55e98c9ef92c75eb629
Jul 24 17:43:00.894723 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:43:00.894701    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c7cb53_638b_4934_9a59_ca69c44a4373.slice/crio-5db70ab1618d685dc38a72414a0756a2b80206339f3ce3a818e6002df9099351 WatchSource:0}: Error finding container 5db70ab1618d685dc38a72414a0756a2b80206339f3ce3a818e6002df9099351: Status 404 returned error can't find the container with id 5db70ab1618d685dc38a72414a0756a2b80206339f3ce3a818e6002df9099351
Jul 24 17:43:00.895576 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:43:00.895113    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cfde4d9_c9a9_4f63_adb4_84c845d4a76a.slice/crio-fd71ac072f58ddda79b29ffbc1cbc7f7ac39340e7bbddcc07a564cd992440305 WatchSource:0}: Error finding container fd71ac072f58ddda79b29ffbc1cbc7f7ac39340e7bbddcc07a564cd992440305: Status 404 returned error can't find the container with id fd71ac072f58ddda79b29ffbc1cbc7f7ac39340e7bbddcc07a564cd992440305
Jul 24 17:43:00.961740 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.961713    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:00.961855 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.961740    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:00.961916 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:00.961866    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmrr7" podUID="4c849bf6-e357-4dca-8005-56fbbcb8198a"
Jul 24 17:43:00.962057 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:00.961981    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g49q" podUID="b4d50741-70f0-4e3b-90aa-f2fcaa98201f"
Jul 24 17:43:00.970507 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.970485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jxx8w" event={"ID":"9ed65d95-64ef-4160-9849-30e2ae5e66d9","Type":"ContainerStarted","Data":"ed383cd30f0ed2d9d3db947132ce2953c8f71a35addc58ae94ff2be6eec36242"}
Jul 24 17:43:00.971674 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.971650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vw29n" event={"ID":"5cfde4d9-c9a9-4f63-adb4-84c845d4a76a","Type":"ContainerStarted","Data":"fd71ac072f58ddda79b29ffbc1cbc7f7ac39340e7bbddcc07a564cd992440305"}
Jul 24 17:43:00.972743 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.972724    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7b5js" event={"ID":"789a3029-0741-46b0-969d-312e2181dbc8","Type":"ContainerStarted","Data":"d05be9e52168da50a0432838a707ed5d6f5057d41d4dd55e98c9ef92c75eb629"}
Jul 24 17:43:00.973745 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.973724    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bnjwn" event={"ID":"65c7cb53-638b-4934-9a59-ca69c44a4373","Type":"ContainerStarted","Data":"5db70ab1618d685dc38a72414a0756a2b80206339f3ce3a818e6002df9099351"}
Jul 24 17:43:00.975095 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.975061    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zsddc" event={"ID":"5ffea99b-424f-4978-8cc0-be6971887c02","Type":"ContainerStarted","Data":"d72ec94c24576c0001871f04762f2d97a65116fb40642cc053088c7695cdebe6"}
Jul 24 17:43:00.976166 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.976139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mn4d8" event={"ID":"afc9b6f4-a944-4228-97db-a0c9bbc70a7f","Type":"ContainerStarted","Data":"4de638b948ef9af7eda262ebede9db50a8963b33cf3096d8afa3d189c71fd241"}
Jul 24 17:43:00.977458 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.977437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46bfx" event={"ID":"b1d776dd-de07-4ba3-94f3-1b2492d09158","Type":"ContainerStarted","Data":"a9debe73fda1dcf57b22acd671c9412075b8e72e6c0b866da0ccf4fe2a8085ef"}
Jul 24 17:43:00.978692 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.978672    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jmdfh" event={"ID":"518b63cc-9a60-48bd-9961-2ee33a833b40","Type":"ContainerStarted","Data":"55e710feb205993703d1ce8a34b254531ebee09181fcb0a51fa835f1a3cbaf89"}
Jul 24 17:43:00.979854 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:00.979834    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf" event={"ID":"29579c17-0f23-4f2e-b0c8-7bb62dbecaec","Type":"ContainerStarted","Data":"bd41d0885bf8f838dce3417818865d1838b88e4ccd10d0b42f8f9a7b373367c9"}
Jul 24 17:43:01.439845 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:01.439054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs\") pod \"network-metrics-daemon-d4gbz\" (UID: \"64e0f517-9969-47f1-8daf-8dfa194ba923\") " pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:01.439845 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:01.439461    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:43:01.439845 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:01.439520    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs podName:64e0f517-9969-47f1-8daf-8dfa194ba923 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:03.439501395 +0000 UTC m=+5.962859871 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs") pod "network-metrics-daemon-d4gbz" (UID: "64e0f517-9969-47f1-8daf-8dfa194ba923") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:43:01.540285 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:01.540210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret\") pod \"global-pull-secret-syncer-mmrr7\" (UID: \"4c849bf6-e357-4dca-8005-56fbbcb8198a\") " pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:01.540462 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:01.540362    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:43:01.540462 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:01.540419    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret podName:4c849bf6-e357-4dca-8005-56fbbcb8198a nodeName:}" failed. No retries permitted until 2026-07-24 17:43:03.540401006 +0000 UTC m=+6.063759464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret") pod "global-pull-secret-syncer-mmrr7" (UID: "4c849bf6-e357-4dca-8005-56fbbcb8198a") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:43:01.641500 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:01.641468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxnc9\" (UniqueName: \"kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9\") pod \"network-check-target-8g49q\" (UID: \"b4d50741-70f0-4e3b-90aa-f2fcaa98201f\") " pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:01.641697 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:01.641670    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:43:01.641790 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:01.641699    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:43:01.641790 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:01.641713    2578 projected.go:196] Error preparing data for projected volume kube-api-access-vxnc9 for pod openshift-network-diagnostics/network-check-target-8g49q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:43:01.641790 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:01.641765    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9 podName:b4d50741-70f0-4e3b-90aa-f2fcaa98201f nodeName:}" failed. No retries permitted until 2026-07-24 17:43:03.641747927 +0000 UTC m=+6.165106386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vxnc9" (UniqueName: "kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9") pod "network-check-target-8g49q" (UID: "b4d50741-70f0-4e3b-90aa-f2fcaa98201f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:43:01.876296 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:01.876185    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 17:37:59 +0000 UTC" deadline="2028-03-15 18:53:03.086757092 +0000 UTC"
Jul 24 17:43:01.876296 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:01.876246    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14401h10m1.210516134s"
Jul 24 17:43:01.964140 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:01.964108    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:01.964319 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:01.964244    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d4gbz" podUID="64e0f517-9969-47f1-8daf-8dfa194ba923"
Jul 24 17:43:01.996418 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:01.995625    2578 generic.go:356] "Generic (PLEG): container finished" podID="b0743ffe14dbbea13ef8e30cf5bd9ed2" containerID="467c4320b58c88f8f2888c258973792d7f0ddb6117da7d47866d4d7f77a91e3c" exitCode=0
Jul 24 17:43:01.996418 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:01.995975    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal" event={"ID":"b0743ffe14dbbea13ef8e30cf5bd9ed2","Type":"ContainerDied","Data":"467c4320b58c88f8f2888c258973792d7f0ddb6117da7d47866d4d7f77a91e3c"}
Jul 24 17:43:02.010183 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:02.009585    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-93.ec2.internal" event={"ID":"b13de3ecc1d0e8f58103000ebb6cab18","Type":"ContainerStarted","Data":"346b53d8f63c82a0532618b4edde92f330032810b9f278be3e2bab1556230176"}
Jul 24 17:43:02.961817 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:02.961174    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:02.961817 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:02.961291    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g49q" podUID="b4d50741-70f0-4e3b-90aa-f2fcaa98201f"
Jul 24 17:43:02.961817 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:02.961670    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:02.961817 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:02.961757    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmrr7" podUID="4c849bf6-e357-4dca-8005-56fbbcb8198a"
Jul 24 17:43:03.031064 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:03.031022    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal" event={"ID":"b0743ffe14dbbea13ef8e30cf5bd9ed2","Type":"ContainerStarted","Data":"2aeab895579823b95dfe11026002ffba2ef4d58462a1be782ece06ed25856c32"}
Jul 24 17:43:03.047170 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:03.047106    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-93.ec2.internal" podStartSLOduration=4.04708718 podStartE2EDuration="4.04708718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:02.029110706 +0000 UTC m=+4.552469186" watchObservedRunningTime="2026-07-24 17:43:03.04708718 +0000 UTC m=+5.570445660"
Jul 24 17:43:03.457115 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:03.457028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs\") pod \"network-metrics-daemon-d4gbz\" (UID: \"64e0f517-9969-47f1-8daf-8dfa194ba923\") " pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:03.457340 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:03.457319    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:43:03.457421 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:03.457396    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs podName:64e0f517-9969-47f1-8daf-8dfa194ba923 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:07.457376898 +0000 UTC m=+9.980735358 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs") pod "network-metrics-daemon-d4gbz" (UID: "64e0f517-9969-47f1-8daf-8dfa194ba923") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:43:03.557799 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:03.557751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret\") pod \"global-pull-secret-syncer-mmrr7\" (UID: \"4c849bf6-e357-4dca-8005-56fbbcb8198a\") " pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:03.557963 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:03.557896    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:43:03.557963 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:03.557961    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret podName:4c849bf6-e357-4dca-8005-56fbbcb8198a nodeName:}" failed. No retries permitted until 2026-07-24 17:43:07.55794148 +0000 UTC m=+10.081299942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret") pod "global-pull-secret-syncer-mmrr7" (UID: "4c849bf6-e357-4dca-8005-56fbbcb8198a") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:43:03.658623 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:03.658582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxnc9\" (UniqueName: \"kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9\") pod \"network-check-target-8g49q\" (UID: \"b4d50741-70f0-4e3b-90aa-f2fcaa98201f\") " pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:03.658826 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:03.658732    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:43:03.658826 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:03.658755    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:43:03.658826 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:03.658767    2578 projected.go:196] Error preparing data for projected volume kube-api-access-vxnc9 for pod openshift-network-diagnostics/network-check-target-8g49q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:43:03.659000 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:03.658842    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9 podName:b4d50741-70f0-4e3b-90aa-f2fcaa98201f nodeName:}" failed. No retries permitted until 2026-07-24 17:43:07.658823355 +0000 UTC m=+10.182181813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vxnc9" (UniqueName: "kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9") pod "network-check-target-8g49q" (UID: "b4d50741-70f0-4e3b-90aa-f2fcaa98201f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:43:03.964616 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:03.964148    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:03.964616 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:03.964283    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d4gbz" podUID="64e0f517-9969-47f1-8daf-8dfa194ba923"
Jul 24 17:43:04.961146 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:04.961064    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:04.961146 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:04.961117    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:04.961388 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:04.961215    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmrr7" podUID="4c849bf6-e357-4dca-8005-56fbbcb8198a"
Jul 24 17:43:04.961388 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:04.961283    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g49q" podUID="b4d50741-70f0-4e3b-90aa-f2fcaa98201f"
Jul 24 17:43:05.962910 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:05.962868    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:05.963326 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:05.963023    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d4gbz" podUID="64e0f517-9969-47f1-8daf-8dfa194ba923"
Jul 24 17:43:06.961516 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:06.961372    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:06.961516 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:06.961419    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:06.961516 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:06.961497    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmrr7" podUID="4c849bf6-e357-4dca-8005-56fbbcb8198a"
Jul 24 17:43:06.961831 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:06.961725    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g49q" podUID="b4d50741-70f0-4e3b-90aa-f2fcaa98201f"
Jul 24 17:43:07.491908 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:07.491867    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs\") pod \"network-metrics-daemon-d4gbz\" (UID: \"64e0f517-9969-47f1-8daf-8dfa194ba923\") " pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:07.492374 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:07.492009    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:43:07.492374 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:07.492071    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs podName:64e0f517-9969-47f1-8daf-8dfa194ba923 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:15.492052842 +0000 UTC m=+18.015411301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs") pod "network-metrics-daemon-d4gbz" (UID: "64e0f517-9969-47f1-8daf-8dfa194ba923") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:43:07.593319 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:07.593252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret\") pod \"global-pull-secret-syncer-mmrr7\" (UID: \"4c849bf6-e357-4dca-8005-56fbbcb8198a\") " pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:07.593495 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:07.593387    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:43:07.593495 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:07.593465    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret podName:4c849bf6-e357-4dca-8005-56fbbcb8198a nodeName:}" failed. No retries permitted until 2026-07-24 17:43:15.593444878 +0000 UTC m=+18.116803362 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret") pod "global-pull-secret-syncer-mmrr7" (UID: "4c849bf6-e357-4dca-8005-56fbbcb8198a") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:43:07.695060 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:07.694472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxnc9\" (UniqueName: \"kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9\") pod \"network-check-target-8g49q\" (UID: \"b4d50741-70f0-4e3b-90aa-f2fcaa98201f\") " pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:07.695060 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:07.694647    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:43:07.695060 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:07.694673    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:43:07.695060 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:07.694686    2578 projected.go:196] Error preparing data for projected volume kube-api-access-vxnc9 for pod openshift-network-diagnostics/network-check-target-8g49q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:43:07.695060 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:07.694747    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9 podName:b4d50741-70f0-4e3b-90aa-f2fcaa98201f nodeName:}" failed. No retries permitted until 2026-07-24 17:43:15.694728469 +0000 UTC m=+18.218086939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vxnc9" (UniqueName: "kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9") pod "network-check-target-8g49q" (UID: "b4d50741-70f0-4e3b-90aa-f2fcaa98201f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:43:07.963582 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:07.963040    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:07.963582 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:07.963192    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d4gbz" podUID="64e0f517-9969-47f1-8daf-8dfa194ba923"
Jul 24 17:43:08.961087 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:08.961053    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:08.961549 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:08.961192    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmrr7" podUID="4c849bf6-e357-4dca-8005-56fbbcb8198a"
Jul 24 17:43:08.961610 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:08.961559    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:08.961661 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:08.961643    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g49q" podUID="b4d50741-70f0-4e3b-90aa-f2fcaa98201f"
Jul 24 17:43:09.961953 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:09.961763    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:09.961953 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:09.961928    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d4gbz" podUID="64e0f517-9969-47f1-8daf-8dfa194ba923"
Jul 24 17:43:10.961420 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:10.961386    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:10.961560 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:10.961386    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:10.961560 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:10.961513    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g49q" podUID="b4d50741-70f0-4e3b-90aa-f2fcaa98201f"
Jul 24 17:43:10.961560 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:10.961551    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmrr7" podUID="4c849bf6-e357-4dca-8005-56fbbcb8198a"
Jul 24 17:43:11.961128 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:11.961095    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:11.961639 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:11.961238    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d4gbz" podUID="64e0f517-9969-47f1-8daf-8dfa194ba923"
Jul 24 17:43:12.961207 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:12.961176    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:12.961207 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:12.961213    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:12.961684 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:12.961305    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g49q" podUID="b4d50741-70f0-4e3b-90aa-f2fcaa98201f"
Jul 24 17:43:12.961684 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:12.961410    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmrr7" podUID="4c849bf6-e357-4dca-8005-56fbbcb8198a"
Jul 24 17:43:13.961449 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:13.961413    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:13.961906 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:13.961568    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d4gbz" podUID="64e0f517-9969-47f1-8daf-8dfa194ba923"
Jul 24 17:43:14.961999 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:14.961963    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:14.961999 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:14.961996    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:14.962497 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:14.962103    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmrr7" podUID="4c849bf6-e357-4dca-8005-56fbbcb8198a"
Jul 24 17:43:14.962497 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:14.962213    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g49q" podUID="b4d50741-70f0-4e3b-90aa-f2fcaa98201f"
Jul 24 17:43:15.553042 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:15.553002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs\") pod \"network-metrics-daemon-d4gbz\" (UID: \"64e0f517-9969-47f1-8daf-8dfa194ba923\") " pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:15.553238 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:15.553149    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:43:15.553238 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:15.553229    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs podName:64e0f517-9969-47f1-8daf-8dfa194ba923 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:31.553207588 +0000 UTC m=+34.076566050 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs") pod "network-metrics-daemon-d4gbz" (UID: "64e0f517-9969-47f1-8daf-8dfa194ba923") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:43:15.654242 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:15.654199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret\") pod \"global-pull-secret-syncer-mmrr7\" (UID: \"4c849bf6-e357-4dca-8005-56fbbcb8198a\") " pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:15.654403 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:15.654364    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:43:15.654458 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:15.654439    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret podName:4c849bf6-e357-4dca-8005-56fbbcb8198a nodeName:}" failed. No retries permitted until 2026-07-24 17:43:31.654419112 +0000 UTC m=+34.177777569 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret") pod "global-pull-secret-syncer-mmrr7" (UID: "4c849bf6-e357-4dca-8005-56fbbcb8198a") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:43:15.754818 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:15.754767    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxnc9\" (UniqueName: \"kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9\") pod \"network-check-target-8g49q\" (UID: \"b4d50741-70f0-4e3b-90aa-f2fcaa98201f\") " pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:15.754998 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:15.754940    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:43:15.754998 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:15.754969    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:43:15.754998 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:15.754984    2578 projected.go:196] Error preparing data for projected volume kube-api-access-vxnc9 for pod openshift-network-diagnostics/network-check-target-8g49q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:43:15.755137 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:15.755048    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9 podName:b4d50741-70f0-4e3b-90aa-f2fcaa98201f nodeName:}" failed. No retries permitted until 2026-07-24 17:43:31.755029499 +0000 UTC m=+34.278387976 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vxnc9" (UniqueName: "kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9") pod "network-check-target-8g49q" (UID: "b4d50741-70f0-4e3b-90aa-f2fcaa98201f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:43:15.961408 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:15.961328    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:15.961546 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:15.961474    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d4gbz" podUID="64e0f517-9969-47f1-8daf-8dfa194ba923"
Jul 24 17:43:16.961301 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:16.961269    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:16.961802 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:16.961269    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:16.961802 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:16.961395    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmrr7" podUID="4c849bf6-e357-4dca-8005-56fbbcb8198a"
Jul 24 17:43:16.961802 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:16.961474    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g49q" podUID="b4d50741-70f0-4e3b-90aa-f2fcaa98201f"
Jul 24 17:43:17.961850 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:17.961812    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:17.962245 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:17.961940    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d4gbz" podUID="64e0f517-9969-47f1-8daf-8dfa194ba923"
Jul 24 17:43:18.961738 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:18.961700    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:18.961931 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:18.961699    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:18.961931 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:18.961843    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g49q" podUID="b4d50741-70f0-4e3b-90aa-f2fcaa98201f"
Jul 24 17:43:18.961931 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:18.961886    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmrr7" podUID="4c849bf6-e357-4dca-8005-56fbbcb8198a"
Jul 24 17:43:19.961849 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:19.961639    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:19.962052 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:19.961966    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d4gbz" podUID="64e0f517-9969-47f1-8daf-8dfa194ba923"
Jul 24 17:43:20.062634 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.062603    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7b5js" event={"ID":"789a3029-0741-46b0-969d-312e2181dbc8","Type":"ContainerStarted","Data":"466020c593219689f798b57ad63c04a1dcc84f9e1bf5af75010eeba104016e85"}
Jul 24 17:43:20.063920 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.063892    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bnjwn" event={"ID":"65c7cb53-638b-4934-9a59-ca69c44a4373","Type":"ContainerStarted","Data":"71f52b75120169cb23e19bc8ed70d3349040400ae4a455759798bd6490598f3b"}
Jul 24 17:43:20.065164 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.065145    2578 generic.go:356] "Generic (PLEG): container finished" podID="5ffea99b-424f-4978-8cc0-be6971887c02" containerID="1169f4c780247c61c9f1900149024fe77ae65b7c20a528c235733ba471d47641" exitCode=0
Jul 24 17:43:20.065244 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.065206    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zsddc" event={"ID":"5ffea99b-424f-4978-8cc0-be6971887c02","Type":"ContainerDied","Data":"1169f4c780247c61c9f1900149024fe77ae65b7c20a528c235733ba471d47641"}
Jul 24 17:43:20.066704 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.066679    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mn4d8" event={"ID":"afc9b6f4-a944-4228-97db-a0c9bbc70a7f","Type":"ContainerStarted","Data":"b1d9e0acff70171a0737245980848d2be7018b2b743f7c4b2b46e1a586af635e"}
Jul 24 17:43:20.069073 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.069053    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46bfx" event={"ID":"b1d776dd-de07-4ba3-94f3-1b2492d09158","Type":"ContainerStarted","Data":"b9177f598b17a2131e8f7a27c3ca0d3bbf7f76811d360c841ae39b5462500d2e"}
Jul 24 17:43:20.069167 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.069110    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46bfx" event={"ID":"b1d776dd-de07-4ba3-94f3-1b2492d09158","Type":"ContainerStarted","Data":"9a512cb7aeb5b60983f5d256ebe05e17773e4bd5921852747d96f801c5322eff"}
Jul 24 17:43:20.069167 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.069144    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46bfx" event={"ID":"b1d776dd-de07-4ba3-94f3-1b2492d09158","Type":"ContainerStarted","Data":"911a082534fd363ee97dce34a43abaf43235f3c34ea5d5ab3a34c27a82d5d8f0"}
Jul 24 17:43:20.069275 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.069184    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46bfx" event={"ID":"b1d776dd-de07-4ba3-94f3-1b2492d09158","Type":"ContainerStarted","Data":"2817bbb3a71751b43ecef47eeab232c6b4dc01a9f1d1c686c7d6a932f3858c88"}
Jul 24 17:43:20.070533 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.070485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jmdfh" event={"ID":"518b63cc-9a60-48bd-9961-2ee33a833b40","Type":"ContainerStarted","Data":"204ae6449c0080b0b15f2c2e208597ab41178bf6e3761695c7784281e9e6c1cd"}
Jul 24 17:43:20.072453 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.072433    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf" event={"ID":"29579c17-0f23-4f2e-b0c8-7bb62dbecaec","Type":"ContainerStarted","Data":"67e0d6d1e45830e5e1417c081c0832c63f591f4b637484a4b28557146df0fed8"}
Jul 24 17:43:20.073696 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.073676    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jxx8w" event={"ID":"9ed65d95-64ef-4160-9849-30e2ae5e66d9","Type":"ContainerStarted","Data":"108a9014b59339f1eba53261b841220f3c19454d165516255baa883f38a13236"}
Jul 24 17:43:20.090103 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.090065    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-93.ec2.internal" podStartSLOduration=21.090053317 podStartE2EDuration="21.090053317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:03.048144514 +0000 UTC m=+5.571502995" watchObservedRunningTime="2026-07-24 17:43:20.090053317 +0000 UTC m=+22.613411794"
Jul 24 17:43:20.108842 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.108806    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7b5js" podStartSLOduration=11.738425378 podStartE2EDuration="22.108766742s" totalImagesPullingTime="10.370341364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:20.108153015 +0000 UTC m=+22.631511492" watchObservedRunningTime="2026-07-24 17:43:20.108766742 +0000 UTC m=+22.632125217"
Jul 24 17:43:20.149383 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.149344    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bnjwn" podStartSLOduration=3.596067871 podStartE2EDuration="22.149329718s" totalImagesPullingTime="18.553261847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:20.149199603 +0000 UTC m=+22.672558085" watchObservedRunningTime="2026-07-24 17:43:20.149329718 +0000 UTC m=+22.672688196"
Jul 24 17:43:20.149702 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.149676    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jxx8w" podStartSLOduration=4.576355282 podStartE2EDuration="23.149668804s" totalImagesPullingTime="18.573313522s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:20.129054475 +0000 UTC m=+22.652412952" watchObservedRunningTime="2026-07-24 17:43:20.149668804 +0000 UTC m=+22.673027327"
Jul 24 17:43:20.168932 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.168900    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mn4d8" podStartSLOduration=3.611544752 podStartE2EDuration="22.168888464s" totalImagesPullingTime="18.557343712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:20.168550614 +0000 UTC m=+22.691909091" watchObservedRunningTime="2026-07-24 17:43:20.168888464 +0000 UTC m=+22.692246941"
Jul 24 17:43:20.191762 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.191723    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jmdfh" podStartSLOduration=3.632470482 podStartE2EDuration="22.191710506s" totalImagesPullingTime="18.559240024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:20.189239255 +0000 UTC m=+22.712597711" watchObservedRunningTime="2026-07-24 17:43:20.191710506 +0000 UTC m=+22.715068984"
Jul 24 17:43:20.962457 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.961989    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:20.962457 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:20.961989    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:20.962457 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:20.962118    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g49q" podUID="b4d50741-70f0-4e3b-90aa-f2fcaa98201f"
Jul 24 17:43:20.962457 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:20.962181    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmrr7" podUID="4c849bf6-e357-4dca-8005-56fbbcb8198a"
Jul 24 17:43:21.041597 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:21.041573    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 17:43:21.077823 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:21.077748    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vw29n" event={"ID":"5cfde4d9-c9a9-4f63-adb4-84c845d4a76a","Type":"ContainerStarted","Data":"27dea0f4bed35600799bbbdfd1db453a2ffbbbd8e30af7044cdad8a6c5310521"}
Jul 24 17:43:21.081418 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:21.081392    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46bfx" event={"ID":"b1d776dd-de07-4ba3-94f3-1b2492d09158","Type":"ContainerStarted","Data":"9a0d0573d59248fa331f482cd99b5b96de4748b9e142e76e99f030d566339bf2"}
Jul 24 17:43:21.081532 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:21.081470    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46bfx" event={"ID":"b1d776dd-de07-4ba3-94f3-1b2492d09158","Type":"ContainerStarted","Data":"f882c76ef06e2f80706d0140c67deb85be7cca5cfc4bac606f0a2eadadd46b02"}
Jul 24 17:43:21.083535 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:21.083512    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf" event={"ID":"29579c17-0f23-4f2e-b0c8-7bb62dbecaec","Type":"ContainerStarted","Data":"9b9d5ccf8ccc1cf12b23aad066f599e125bf4e179f7b0f8a798ac510edd72676"}
Jul 24 17:43:21.094362 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:21.094325    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vw29n" podStartSLOduration=4.54184906 podStartE2EDuration="23.094312095s" totalImagesPullingTime="18.552463035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:21.093636931 +0000 UTC m=+23.616995435" watchObservedRunningTime="2026-07-24 17:43:21.094312095 +0000 UTC m=+23.617670573"
Jul 24 17:43:21.880581 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:21.880474    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T17:43:21.041593808Z","UUID":"322ae84c-5fb3-434b-b713-f9e88c897cd5","Handler":null,"Name":"","Endpoint":""}
Jul 24 17:43:21.884113 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:21.884086    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 17:43:21.884260 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:21.884140    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 17:43:21.962186 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:21.962153    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:21.962338 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:21.962297    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d4gbz" podUID="64e0f517-9969-47f1-8daf-8dfa194ba923"
Jul 24 17:43:22.961266 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:22.961235    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:22.961729 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:22.961364    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmrr7" podUID="4c849bf6-e357-4dca-8005-56fbbcb8198a"
Jul 24 17:43:22.961729 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:22.961422    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:22.961729 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:22.961546    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g49q" podUID="b4d50741-70f0-4e3b-90aa-f2fcaa98201f"
Jul 24 17:43:23.092254 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:23.092050    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46bfx" event={"ID":"b1d776dd-de07-4ba3-94f3-1b2492d09158","Type":"ContainerStarted","Data":"e6fa8a8a66b1b557b675252ff8aa66b59e7600dcefac285a61303f84f29306dd"}
Jul 24 17:43:23.097077 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:23.097047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf" event={"ID":"29579c17-0f23-4f2e-b0c8-7bb62dbecaec","Type":"ContainerStarted","Data":"fef00ba27f2dbab8323b28932edfa0d1f3f299b1698c61a031393732c899e62a"}
Jul 24 17:43:23.115157 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:23.115106    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8w2kf" podStartSLOduration=4.298340382 podStartE2EDuration="25.115092653s" totalImagesPullingTime="20.816752271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:23.114567402 +0000 UTC m=+25.637925881" watchObservedRunningTime="2026-07-24 17:43:23.115092653 +0000 UTC m=+25.638451130"
Jul 24 17:43:23.961551 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:23.961517    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:23.962000 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:23.961656    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d4gbz" podUID="64e0f517-9969-47f1-8daf-8dfa194ba923"
Jul 24 17:43:24.864510 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:24.864273    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7b5js"
Jul 24 17:43:24.864947 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:24.864930    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7b5js"
Jul 24 17:43:24.961368 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:24.961275    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:24.961368 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:24.961275    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:24.961368 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:24.961367    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g49q" podUID="b4d50741-70f0-4e3b-90aa-f2fcaa98201f"
Jul 24 17:43:24.961587 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:24.961438    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmrr7" podUID="4c849bf6-e357-4dca-8005-56fbbcb8198a"
Jul 24 17:43:25.103003 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:25.102970    2578 generic.go:356] "Generic (PLEG): container finished" podID="5ffea99b-424f-4978-8cc0-be6971887c02" containerID="42a70efc4d09da36ee158cb0ae310d7f8d605518b0df6e5639821b551ef2f869" exitCode=0
Jul 24 17:43:25.103176 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:25.103048    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zsddc" event={"ID":"5ffea99b-424f-4978-8cc0-be6971887c02","Type":"ContainerDied","Data":"42a70efc4d09da36ee158cb0ae310d7f8d605518b0df6e5639821b551ef2f869"}
Jul 24 17:43:25.106739 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:25.106717    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46bfx" event={"ID":"b1d776dd-de07-4ba3-94f3-1b2492d09158","Type":"ContainerStarted","Data":"5a628ebc115bf16a2320a5a1e5c51828a725f47b4a0eb4c64b1ccbc565702368"}
Jul 24 17:43:25.107022 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:25.107006    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7b5js"
Jul 24 17:43:25.107412 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:25.107398    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7b5js"
Jul 24 17:43:25.172251 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:25.172210    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-46bfx" podStartSLOduration=8.486620746 podStartE2EDuration="27.17219581s" totalImagesPullingTime="18.685575064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:25.170925863 +0000 UTC m=+27.694284340" watchObservedRunningTime="2026-07-24 17:43:25.17219581 +0000 UTC m=+27.695554287"
Jul 24 17:43:25.961837 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:25.961800    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:25.962282 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:25.961938    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d4gbz" podUID="64e0f517-9969-47f1-8daf-8dfa194ba923"
Jul 24 17:43:26.109105 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:26.109034    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:26.109105 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:26.109102    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:26.109306 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:26.109131    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:26.123974 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:26.123952    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:26.124106 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:26.124048    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:26.635062 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:26.634990    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8g49q"]
Jul 24 17:43:26.635314 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:26.635103    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:26.635314 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:26.635186    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g49q" podUID="b4d50741-70f0-4e3b-90aa-f2fcaa98201f"
Jul 24 17:43:26.637831 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:26.637803    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mmrr7"]
Jul 24 17:43:26.637971 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:26.637915    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:26.638045 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:26.638014    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmrr7" podUID="4c849bf6-e357-4dca-8005-56fbbcb8198a"
Jul 24 17:43:26.638496 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:26.638475    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d4gbz"]
Jul 24 17:43:26.638580 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:26.638566    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:26.638695 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:26.638674    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d4gbz" podUID="64e0f517-9969-47f1-8daf-8dfa194ba923"
Jul 24 17:43:27.112564 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:27.112338    2578 generic.go:356] "Generic (PLEG): container finished" podID="5ffea99b-424f-4978-8cc0-be6971887c02" containerID="e11310a2374e087a4bbd822631f0f997ea47b15cd040e1480dedbec8c16e7cfe" exitCode=0
Jul 24 17:43:27.112564 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:27.112410    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zsddc" event={"ID":"5ffea99b-424f-4978-8cc0-be6971887c02","Type":"ContainerDied","Data":"e11310a2374e087a4bbd822631f0f997ea47b15cd040e1480dedbec8c16e7cfe"}
Jul 24 17:43:27.962702 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:27.962669    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:27.962873 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:27.962805    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:27.962956 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:27.962934    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmrr7" podUID="4c849bf6-e357-4dca-8005-56fbbcb8198a"
Jul 24 17:43:27.963006 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:27.962796    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d4gbz" podUID="64e0f517-9969-47f1-8daf-8dfa194ba923"
Jul 24 17:43:28.961727 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:28.961648    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:28.962101 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:28.961754    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g49q" podUID="b4d50741-70f0-4e3b-90aa-f2fcaa98201f"
Jul 24 17:43:29.119316 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:29.119286    2578 generic.go:356] "Generic (PLEG): container finished" podID="5ffea99b-424f-4978-8cc0-be6971887c02" containerID="6ec5703bd27da685414be02484cd49e4dba3803621405a4417aeeb722d5f59f6" exitCode=0
Jul 24 17:43:29.119463 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:29.119349    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zsddc" event={"ID":"5ffea99b-424f-4978-8cc0-be6971887c02","Type":"ContainerDied","Data":"6ec5703bd27da685414be02484cd49e4dba3803621405a4417aeeb722d5f59f6"}
Jul 24 17:43:29.962125 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:29.962091    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:29.962902 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:29.962206    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmrr7" podUID="4c849bf6-e357-4dca-8005-56fbbcb8198a"
Jul 24 17:43:29.962902 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:29.962263    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:29.962902 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:29.962399    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d4gbz" podUID="64e0f517-9969-47f1-8daf-8dfa194ba923"
Jul 24 17:43:30.961532 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:30.961495    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:30.961709 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:30.961605    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g49q" podUID="b4d50741-70f0-4e3b-90aa-f2fcaa98201f"
Jul 24 17:43:31.571428 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:31.571396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs\") pod \"network-metrics-daemon-d4gbz\" (UID: \"64e0f517-9969-47f1-8daf-8dfa194ba923\") " pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:31.571932 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:31.571508    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:43:31.571932 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:31.571567    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs podName:64e0f517-9969-47f1-8daf-8dfa194ba923 nodeName:}" failed. No retries permitted until 2026-07-24 17:44:03.571551622 +0000 UTC m=+66.094910077 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs") pod "network-metrics-daemon-d4gbz" (UID: "64e0f517-9969-47f1-8daf-8dfa194ba923") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:43:31.672236 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:31.672200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret\") pod \"global-pull-secret-syncer-mmrr7\" (UID: \"4c849bf6-e357-4dca-8005-56fbbcb8198a\") " pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:31.672397 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:31.672357    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:43:31.672459 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:31.672427    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret podName:4c849bf6-e357-4dca-8005-56fbbcb8198a nodeName:}" failed. No retries permitted until 2026-07-24 17:44:03.672410251 +0000 UTC m=+66.195768710 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret") pod "global-pull-secret-syncer-mmrr7" (UID: "4c849bf6-e357-4dca-8005-56fbbcb8198a") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:43:31.772944 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:31.772914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxnc9\" (UniqueName: \"kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9\") pod \"network-check-target-8g49q\" (UID: \"b4d50741-70f0-4e3b-90aa-f2fcaa98201f\") " pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:31.773111 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:31.773093    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:43:31.773162 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:31.773116    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:43:31.773162 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:31.773127    2578 projected.go:196] Error preparing data for projected volume kube-api-access-vxnc9 for pod openshift-network-diagnostics/network-check-target-8g49q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:43:31.773226 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:31.773182    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9 podName:b4d50741-70f0-4e3b-90aa-f2fcaa98201f nodeName:}" failed. No retries permitted until 2026-07-24 17:44:03.773168514 +0000 UTC m=+66.296526989 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vxnc9" (UniqueName: "kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9") pod "network-check-target-8g49q" (UID: "b4d50741-70f0-4e3b-90aa-f2fcaa98201f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:43:31.964952 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:31.964878    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:31.965116 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:31.964878    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:31.965116 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:31.965009    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d4gbz" podUID="64e0f517-9969-47f1-8daf-8dfa194ba923"
Jul 24 17:43:31.965116 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:31.965079    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmrr7" podUID="4c849bf6-e357-4dca-8005-56fbbcb8198a"
Jul 24 17:43:32.350510 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.350480    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-93.ec2.internal" event="NodeReady"
Jul 24 17:43:32.350665 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.350623    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 17:43:32.487002 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.486968    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9jtqg"]
Jul 24 17:43:32.489259 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.489238    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7df8c856dc-jqsbv"]
Jul 24 17:43:32.489382 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.489364    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:43:32.490968 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.490944    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x95l7"]
Jul 24 17:43:32.491088 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.491070    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.492950 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.492927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x95l7"
Jul 24 17:43:32.493051 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:32.492963    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"dns-default-9jtqg\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" podUID="8929ae32-e84a-4a3b-b7ad-5ee3f7575298" pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:43:32.493051 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:32.492983    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"dns-dockercfg-5kpm4\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5kpm4\"" type="*v1.Secret"
Jul 24 17:43:32.493655 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:32.493628    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"dns-default\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap"
Jul 24 17:43:32.494076 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:32.494053    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"image-registry-tls\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" type="*v1.Secret"
Jul 24 17:43:32.494076 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:32.494054    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"dns-default-metrics-tls\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret"
Jul 24 17:43:32.494292 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:32.494271    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"registry-dockercfg-vh2n6\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vh2n6\"" type="*v1.Secret"
Jul 24 17:43:32.494345 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:32.494320    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"installation-pull-secrets\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" type="*v1.Secret"
Jul 24 17:43:32.494528 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:32.494504    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"trusted-ca\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" type="*v1.ConfigMap"
Jul 24 17:43:32.494977 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:32.494954    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"image-registry-7df8c856dc-jqsbv\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" podUID="33baa17d-da25-4f28-881f-55035131c237" pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.496901 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:32.496871    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"image-registry-private-configuration\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" type="*v1.Secret"
Jul 24 17:43:32.497851 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:32.497811    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"canary-serving-cert\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret"
Jul 24 17:43:32.498365 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:32.498056    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-t2cd6\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-t2cd6\"" type="*v1.Secret"
Jul 24 17:43:32.498365 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:32.498065    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 24 17:43:32.498365 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:32.498076    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 24 17:43:32.498544 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:32.498464    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"ingress-canary-x95l7\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" podUID="9f1aee3c-1324-4eaa-8201-95c26602d6c1" pod="openshift-ingress-canary/ingress-canary-x95l7"
Jul 24 17:43:32.534970 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.534945    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9jtqg"]
Jul 24 17:43:32.537848 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.537825    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x95l7"]
Jul 24 17:43:32.542206 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.542152    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7df8c856dc-jqsbv"]
Jul 24 17:43:32.579372 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.579346    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvn2p\" (UniqueName: \"kubernetes.io/projected/9f1aee3c-1324-4eaa-8201-95c26602d6c1-kube-api-access-wvn2p\") pod \"ingress-canary-x95l7\" (UID: \"9f1aee3c-1324-4eaa-8201-95c26602d6c1\") " pod="openshift-ingress-canary/ingress-canary-x95l7"
Jul 24 17:43:32.579745 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.579382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wtms\" (UniqueName: \"kubernetes.io/projected/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-kube-api-access-4wtms\") pod \"dns-default-9jtqg\" (UID: \"8929ae32-e84a-4a3b-b7ad-5ee3f7575298\") " pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:43:32.579745 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.579401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33baa17d-da25-4f28-881f-55035131c237-ca-trust-extracted\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.579745 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.579426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33baa17d-da25-4f28-881f-55035131c237-trusted-ca\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.579745 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.579505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-config-volume\") pod \"dns-default-9jtqg\" (UID: \"8929ae32-e84a-4a3b-b7ad-5ee3f7575298\") " pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:43:32.579745 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.579520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-tmp-dir\") pod \"dns-default-9jtqg\" (UID: \"8929ae32-e84a-4a3b-b7ad-5ee3f7575298\") " pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:43:32.579745 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.579571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f1aee3c-1324-4eaa-8201-95c26602d6c1-cert\") pod \"ingress-canary-x95l7\" (UID: \"9f1aee3c-1324-4eaa-8201-95c26602d6c1\") " pod="openshift-ingress-canary/ingress-canary-x95l7"
Jul 24 17:43:32.579745 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.579589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33baa17d-da25-4f28-881f-55035131c237-registry-certificates\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.579745 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.579613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33baa17d-da25-4f28-881f-55035131c237-installation-pull-secrets\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.579745 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.579738    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls\") pod \"dns-default-9jtqg\" (UID: \"8929ae32-e84a-4a3b-b7ad-5ee3f7575298\") " pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:43:32.580130 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.579773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33baa17d-da25-4f28-881f-55035131c237-image-registry-private-configuration\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.580130 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.579812    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.580130 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.579835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-bound-sa-token\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.580130 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.579860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzhkj\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-kube-api-access-zzhkj\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.680822 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.680726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls\") pod \"dns-default-9jtqg\" (UID: \"8929ae32-e84a-4a3b-b7ad-5ee3f7575298\") " pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:43:32.680822 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.680769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33baa17d-da25-4f28-881f-55035131c237-image-registry-private-configuration\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.681027 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.680908    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.681027 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.680946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-bound-sa-token\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.681027 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.680969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzhkj\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-kube-api-access-zzhkj\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.681027 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.680999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvn2p\" (UniqueName: \"kubernetes.io/projected/9f1aee3c-1324-4eaa-8201-95c26602d6c1-kube-api-access-wvn2p\") pod \"ingress-canary-x95l7\" (UID: \"9f1aee3c-1324-4eaa-8201-95c26602d6c1\") " pod="openshift-ingress-canary/ingress-canary-x95l7"
Jul 24 17:43:32.681222 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.681034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wtms\" (UniqueName: \"kubernetes.io/projected/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-kube-api-access-4wtms\") pod \"dns-default-9jtqg\" (UID: \"8929ae32-e84a-4a3b-b7ad-5ee3f7575298\") " pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:43:32.681222 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.681062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33baa17d-da25-4f28-881f-55035131c237-ca-trust-extracted\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.681222 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.681087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33baa17d-da25-4f28-881f-55035131c237-trusted-ca\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.681222 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.681135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-config-volume\") pod \"dns-default-9jtqg\" (UID: \"8929ae32-e84a-4a3b-b7ad-5ee3f7575298\") " pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:43:32.681222 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.681160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-tmp-dir\") pod \"dns-default-9jtqg\" (UID: \"8929ae32-e84a-4a3b-b7ad-5ee3f7575298\") " pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:43:32.681222 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.681200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f1aee3c-1324-4eaa-8201-95c26602d6c1-cert\") pod \"ingress-canary-x95l7\" (UID: \"9f1aee3c-1324-4eaa-8201-95c26602d6c1\") " pod="openshift-ingress-canary/ingress-canary-x95l7"
Jul 24 17:43:32.681508 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.681227    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33baa17d-da25-4f28-881f-55035131c237-registry-certificates\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.681508 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.681257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33baa17d-da25-4f28-881f-55035131c237-installation-pull-secrets\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.681508 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.681493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33baa17d-da25-4f28-881f-55035131c237-ca-trust-extracted\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.681614 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.681579    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-tmp-dir\") pod \"dns-default-9jtqg\" (UID: \"8929ae32-e84a-4a3b-b7ad-5ee3f7575298\") " pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:43:32.681955 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.681929    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33baa17d-da25-4f28-881f-55035131c237-registry-certificates\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.700248 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.700223    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wtms\" (UniqueName: \"kubernetes.io/projected/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-kube-api-access-4wtms\") pod \"dns-default-9jtqg\" (UID: \"8929ae32-e84a-4a3b-b7ad-5ee3f7575298\") " pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:43:32.700373 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.700357    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzhkj\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-kube-api-access-zzhkj\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.701305 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.701277    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-bound-sa-token\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:32.961793 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.961703    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:43:32.965379 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.965353    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 17:43:32.965503 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.965426    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 17:43:32.965566 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:32.965426    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9txdv\""
Jul 24 17:43:33.301642 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:33.301618    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5kpm4\""
Jul 24 17:43:33.309836 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:33.309816    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 17:43:33.339979 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:33.339954    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 17:43:33.353023 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:33.352999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvn2p\" (UniqueName: \"kubernetes.io/projected/9f1aee3c-1324-4eaa-8201-95c26602d6c1-kube-api-access-wvn2p\") pod \"ingress-canary-x95l7\" (UID: \"9f1aee3c-1324-4eaa-8201-95c26602d6c1\") " pod="openshift-ingress-canary/ingress-canary-x95l7"
Jul 24 17:43:33.387569 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:33.387544    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vh2n6\""
Jul 24 17:43:33.613510 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:33.613484    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 24 17:43:33.624001 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:33.623973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33baa17d-da25-4f28-881f-55035131c237-installation-pull-secrets\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:33.681823 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:33.681795    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: failed to sync secret cache: timed out waiting for the condition
Jul 24 17:43:33.681823 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:33.681810    2578 configmap.go:193] Couldn't get configMap openshift-image-registry/trusted-ca: failed to sync configmap cache: timed out waiting for the condition
Jul 24 17:43:33.681823 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:33.681824    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: failed to sync secret cache: timed out waiting for the condition
Jul 24 17:43:33.682007 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:33.681842    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df8c856dc-jqsbv: failed to sync secret cache: timed out waiting for the condition
Jul 24 17:43:33.682007 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:33.681847    2578 secret.go:189] Couldn't get secret openshift-image-registry/image-registry-private-configuration: failed to sync secret cache: timed out waiting for the condition
Jul 24 17:43:33.682007 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:33.681817    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: failed to sync secret cache: timed out waiting for the condition
Jul 24 17:43:33.682007 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:33.681865    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls podName:8929ae32-e84a-4a3b-b7ad-5ee3f7575298 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:34.181850385 +0000 UTC m=+36.705208840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls") pod "dns-default-9jtqg" (UID: "8929ae32-e84a-4a3b-b7ad-5ee3f7575298") : failed to sync secret cache: timed out waiting for the condition
Jul 24 17:43:33.682007 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:33.681795    2578 configmap.go:193] Couldn't get configMap openshift-dns/dns-default: failed to sync configmap cache: timed out waiting for the condition
Jul 24 17:43:33.682007 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:33.681889    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f1aee3c-1324-4eaa-8201-95c26602d6c1-cert podName:9f1aee3c-1324-4eaa-8201-95c26602d6c1 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:34.181874984 +0000 UTC m=+36.705233464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9f1aee3c-1324-4eaa-8201-95c26602d6c1-cert") pod "ingress-canary-x95l7" (UID: "9f1aee3c-1324-4eaa-8201-95c26602d6c1") : failed to sync secret cache: timed out waiting for the condition
Jul 24 17:43:33.682007 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:33.681901    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/33baa17d-da25-4f28-881f-55035131c237-trusted-ca podName:33baa17d-da25-4f28-881f-55035131c237 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:34.181895339 +0000 UTC m=+36.705253794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca" (UniqueName: "kubernetes.io/configmap/33baa17d-da25-4f28-881f-55035131c237-trusted-ca") pod "image-registry-7df8c856dc-jqsbv" (UID: "33baa17d-da25-4f28-881f-55035131c237") : failed to sync configmap cache: timed out waiting for the condition
Jul 24 17:43:33.682007 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:33.681913    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls podName:33baa17d-da25-4f28-881f-55035131c237 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:34.181907608 +0000 UTC m=+36.705266063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls") pod "image-registry-7df8c856dc-jqsbv" (UID: "33baa17d-da25-4f28-881f-55035131c237") : failed to sync secret cache: timed out waiting for the condition
Jul 24 17:43:33.682007 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:33.681928    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33baa17d-da25-4f28-881f-55035131c237-image-registry-private-configuration podName:33baa17d-da25-4f28-881f-55035131c237 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:34.181921689 +0000 UTC m=+36.705280145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "image-registry-private-configuration" (UniqueName: "kubernetes.io/secret/33baa17d-da25-4f28-881f-55035131c237-image-registry-private-configuration") pod "image-registry-7df8c856dc-jqsbv" (UID: "33baa17d-da25-4f28-881f-55035131c237") : failed to sync secret cache: timed out waiting for the condition
Jul 24 17:43:33.682007 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:33.681942    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-config-volume podName:8929ae32-e84a-4a3b-b7ad-5ee3f7575298 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:34.181937264 +0000 UTC m=+36.705295720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "config-volume" (UniqueName: "kubernetes.io/configmap/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-config-volume") pod "dns-default-9jtqg" (UID: "8929ae32-e84a-4a3b-b7ad-5ee3f7575298") : failed to sync configmap cache: timed out waiting for the condition
Jul 24 17:43:33.704672 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:33.704650    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 17:43:33.740240 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:33.740197    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 17:43:33.754217 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:33.754198    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-t2cd6\""
Jul 24 17:43:33.846568 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:33.846538    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 24 17:43:33.961315 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:33.961284    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:43:33.961457 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:33.961317    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:43:33.968582 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:33.968564    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 17:43:33.968582 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:33.968579    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 17:43:33.968740 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:33.968709    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pdhgn\""
Jul 24 17:43:34.015203 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:34.015173    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 24 17:43:34.080662 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:34.080635    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 17:43:34.082417 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:34.082401    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 24 17:43:34.133216 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:34.133141    2578 generic.go:356] "Generic (PLEG): container finished" podID="5ffea99b-424f-4978-8cc0-be6971887c02" containerID="8d9aacc2ff12ca1ec46f33ae7a7acde7f958a5474fc17f3f7c059e16e2170c27" exitCode=0
Jul 24 17:43:34.133216 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:34.133207    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zsddc" event={"ID":"5ffea99b-424f-4978-8cc0-be6971887c02","Type":"ContainerDied","Data":"8d9aacc2ff12ca1ec46f33ae7a7acde7f958a5474fc17f3f7c059e16e2170c27"}
Jul 24 17:43:34.193148 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:34.193024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33baa17d-da25-4f28-881f-55035131c237-trusted-ca\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:34.193148 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:34.193073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-config-volume\") pod \"dns-default-9jtqg\" (UID: \"8929ae32-e84a-4a3b-b7ad-5ee3f7575298\") " pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:43:34.193148 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:34.193111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f1aee3c-1324-4eaa-8201-95c26602d6c1-cert\") pod \"ingress-canary-x95l7\" (UID: \"9f1aee3c-1324-4eaa-8201-95c26602d6c1\") " pod="openshift-ingress-canary/ingress-canary-x95l7"
Jul 24 17:43:34.193323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:34.193180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls\") pod \"dns-default-9jtqg\" (UID: \"8929ae32-e84a-4a3b-b7ad-5ee3f7575298\") " pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:43:34.193323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:34.193211    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33baa17d-da25-4f28-881f-55035131c237-image-registry-private-configuration\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:34.193323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:34.193240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:34.193470 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:34.193373    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 17:43:34.193470 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:34.193389    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df8c856dc-jqsbv: secret "image-registry-tls" not found
Jul 24 17:43:34.193470 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:34.193453    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls podName:33baa17d-da25-4f28-881f-55035131c237 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:35.193435511 +0000 UTC m=+37.716793986 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls") pod "image-registry-7df8c856dc-jqsbv" (UID: "33baa17d-da25-4f28-881f-55035131c237") : secret "image-registry-tls" not found
Jul 24 17:43:34.193635 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:34.193568    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:43:34.193635 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:34.193635    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls podName:8929ae32-e84a-4a3b-b7ad-5ee3f7575298 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:35.193618188 +0000 UTC m=+37.716976646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls") pod "dns-default-9jtqg" (UID: "8929ae32-e84a-4a3b-b7ad-5ee3f7575298") : secret "dns-default-metrics-tls" not found
Jul 24 17:43:34.193742 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:34.193699    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-config-volume\") pod \"dns-default-9jtqg\" (UID: \"8929ae32-e84a-4a3b-b7ad-5ee3f7575298\") " pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:43:34.194128 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:34.194108    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33baa17d-da25-4f28-881f-55035131c237-trusted-ca\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:34.195945 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:34.195922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33baa17d-da25-4f28-881f-55035131c237-image-registry-private-configuration\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:34.196385 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:34.196370    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f1aee3c-1324-4eaa-8201-95c26602d6c1-cert\") pod \"ingress-canary-x95l7\" (UID: \"9f1aee3c-1324-4eaa-8201-95c26602d6c1\") " pod="openshift-ingress-canary/ingress-canary-x95l7"
Jul 24 17:43:34.314530 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:34.314491    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x95l7"
Jul 24 17:43:34.446456 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:34.446429    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x95l7"]
Jul 24 17:43:34.447091 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:43:34.447067    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1aee3c_1324_4eaa_8201_95c26602d6c1.slice/crio-5fbe95898517de23d97a616c3c0b5d4e3c201a10ee0aebeec5f29fd041016ee1 WatchSource:0}: Error finding container 5fbe95898517de23d97a616c3c0b5d4e3c201a10ee0aebeec5f29fd041016ee1: Status 404 returned error can't find the container with id 5fbe95898517de23d97a616c3c0b5d4e3c201a10ee0aebeec5f29fd041016ee1
Jul 24 17:43:35.138540 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:35.138506    2578 generic.go:356] "Generic (PLEG): container finished" podID="5ffea99b-424f-4978-8cc0-be6971887c02" containerID="8b8b2b680428acce760639eafb44219b49824c500b2ff37e604e880ad90cae3d" exitCode=0
Jul 24 17:43:35.139006 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:35.138582    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zsddc" event={"ID":"5ffea99b-424f-4978-8cc0-be6971887c02","Type":"ContainerDied","Data":"8b8b2b680428acce760639eafb44219b49824c500b2ff37e604e880ad90cae3d"}
Jul 24 17:43:35.139746 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:35.139724    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x95l7" event={"ID":"9f1aee3c-1324-4eaa-8201-95c26602d6c1","Type":"ContainerStarted","Data":"5fbe95898517de23d97a616c3c0b5d4e3c201a10ee0aebeec5f29fd041016ee1"}
Jul 24 17:43:35.201229 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:35.201025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls\") pod \"dns-default-9jtqg\" (UID: \"8929ae32-e84a-4a3b-b7ad-5ee3f7575298\") " pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:43:35.201366 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:35.201257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:35.201366 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:35.201193    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:43:35.201366 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:35.201366    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls podName:8929ae32-e84a-4a3b-b7ad-5ee3f7575298 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:37.201346604 +0000 UTC m=+39.724705084 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls") pod "dns-default-9jtqg" (UID: "8929ae32-e84a-4a3b-b7ad-5ee3f7575298") : secret "dns-default-metrics-tls" not found
Jul 24 17:43:35.201534 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:35.201384    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 17:43:35.201534 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:35.201399    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df8c856dc-jqsbv: secret "image-registry-tls" not found
Jul 24 17:43:35.201534 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:35.201454    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls podName:33baa17d-da25-4f28-881f-55035131c237 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:37.201436519 +0000 UTC m=+39.724794992 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls") pod "image-registry-7df8c856dc-jqsbv" (UID: "33baa17d-da25-4f28-881f-55035131c237") : secret "image-registry-tls" not found
Jul 24 17:43:35.705526 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:35.705491    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:35.797562 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:35.797534    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:36.146014 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:36.145978    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zsddc" event={"ID":"5ffea99b-424f-4978-8cc0-be6971887c02","Type":"ContainerStarted","Data":"0ffe2d5e7607ce70c432e8356d12abcc5dfb1987b03637625e3f77774d3df076"}
Jul 24 17:43:36.173244 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:36.173196    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zsddc" podStartSLOduration=9.8681958 podStartE2EDuration="39.173180695s" totalImagesPullingTime="29.304984895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:36.171175407 +0000 UTC m=+38.694533884" watchObservedRunningTime="2026-07-24 17:43:36.173180695 +0000 UTC m=+38.696539172"
Jul 24 17:43:36.293089 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:36.293061    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:36.394176 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:36.394140    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:37.065655 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:37.065624    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:37.132214 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:37.132184    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:37.220537 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:37.220504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls\") pod \"dns-default-9jtqg\" (UID: \"8929ae32-e84a-4a3b-b7ad-5ee3f7575298\") " pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:43:37.220971 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:37.220544    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:37.220971 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:37.220636    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:43:37.220971 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:37.220699    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls podName:8929ae32-e84a-4a3b-b7ad-5ee3f7575298 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:41.220683211 +0000 UTC m=+43.744041670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls") pod "dns-default-9jtqg" (UID: "8929ae32-e84a-4a3b-b7ad-5ee3f7575298") : secret "dns-default-metrics-tls" not found
Jul 24 17:43:37.220971 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:37.220637    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 17:43:37.220971 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:37.220721    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df8c856dc-jqsbv: secret "image-registry-tls" not found
Jul 24 17:43:37.220971 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:37.220748    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls podName:33baa17d-da25-4f28-881f-55035131c237 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:41.220741386 +0000 UTC m=+43.744099858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls") pod "image-registry-7df8c856dc-jqsbv" (UID: "33baa17d-da25-4f28-881f-55035131c237") : secret "image-registry-tls" not found
Jul 24 17:43:38.151185 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:38.151155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x95l7" event={"ID":"9f1aee3c-1324-4eaa-8201-95c26602d6c1","Type":"ContainerStarted","Data":"ac1b21d063feae7430a1e6896f8fb64c291431f91e1ecf59935a06016465e75d"}
Jul 24 17:43:38.169168 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:38.169124    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x95l7" podStartSLOduration=3.206217454 podStartE2EDuration="6.169109077s" totalImagesPullingTime="2.962891623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:38.167884733 +0000 UTC m=+40.691243202" watchObservedRunningTime="2026-07-24 17:43:38.169109077 +0000 UTC m=+40.692467555"
Jul 24 17:43:41.249439 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:41.249401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls\") pod \"dns-default-9jtqg\" (UID: \"8929ae32-e84a-4a3b-b7ad-5ee3f7575298\") " pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:43:41.249439 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:41.249442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:41.249889 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:41.249554    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:43:41.249889 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:41.249566    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 17:43:41.249889 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:41.249579    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df8c856dc-jqsbv: secret "image-registry-tls" not found
Jul 24 17:43:41.249889 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:41.249619    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls podName:8929ae32-e84a-4a3b-b7ad-5ee3f7575298 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:49.249604326 +0000 UTC m=+51.772962782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls") pod "dns-default-9jtqg" (UID: "8929ae32-e84a-4a3b-b7ad-5ee3f7575298") : secret "dns-default-metrics-tls" not found
Jul 24 17:43:41.249889 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:41.249634    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls podName:33baa17d-da25-4f28-881f-55035131c237 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:49.249627017 +0000 UTC m=+51.772985473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls") pod "image-registry-7df8c856dc-jqsbv" (UID: "33baa17d-da25-4f28-881f-55035131c237") : secret "image-registry-tls" not found
Jul 24 17:43:42.911021 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:42.910987    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:43.056901 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:43.056858    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:43.170801 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:43.170705    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:43.886593 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:43.886540    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:43.987214 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:43.987180    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:48.071289 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:48.071258    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:48.142709 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:48.142671    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:48.990007 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:48.989972    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pfqcd"]
Jul 24 17:43:49.005643 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.005617    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:49.008623 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.008597    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 17:43:49.009506 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.009480    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pfqcd"]
Jul 24 17:43:49.009892 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.009874    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 17:43:49.009892 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.009877    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 17:43:49.010067 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.009880    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 17:43:49.010067 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.009966    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cg7vk\""
Jul 24 17:43:49.023132 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.023110    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:49.105071 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.105040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:49.105428 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.105081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:49.105428 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.105185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8d0a6d2-c541-427e-b1e2-48fa4db65735-crio-socket\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:49.105428 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.105208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8d0a6d2-c541-427e-b1e2-48fa4db65735-data-volume\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:49.105428 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.105228    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqm6s\" (UniqueName: \"kubernetes.io/projected/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-api-access-hqm6s\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:49.200917 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.200885    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:49.205965 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.205946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:49.206062 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.205974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:49.206062 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.206007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8d0a6d2-c541-427e-b1e2-48fa4db65735-crio-socket\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:49.206062 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.206028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8d0a6d2-c541-427e-b1e2-48fa4db65735-data-volume\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:49.206062 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.206053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqm6s\" (UniqueName: \"kubernetes.io/projected/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-api-access-hqm6s\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:49.206226 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:49.206068    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 17:43:49.206226 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:49.206114    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 17:43:49.206226 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:49.206141    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-rbac-proxy-cm podName:d8d0a6d2-c541-427e-b1e2-48fa4db65735 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:49.706120893 +0000 UTC m=+52.229479350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pfqcd" (UID: "d8d0a6d2-c541-427e-b1e2-48fa4db65735") : configmap "kube-rbac-proxy" not found
Jul 24 17:43:49.206226 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:49.206177    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls podName:d8d0a6d2-c541-427e-b1e2-48fa4db65735 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:49.706161854 +0000 UTC m=+52.229520331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pfqcd" (UID: "d8d0a6d2-c541-427e-b1e2-48fa4db65735") : secret "insights-runtime-extractor-tls" not found
Jul 24 17:43:49.206366 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.206292    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8d0a6d2-c541-427e-b1e2-48fa4db65735-crio-socket\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:49.206366 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.206316    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8d0a6d2-c541-427e-b1e2-48fa4db65735-data-volume\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:49.222928 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.222897    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqm6s\" (UniqueName: \"kubernetes.io/projected/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-api-access-hqm6s\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:49.288042 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.287962    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:49.307009 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.306977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls\") pod \"dns-default-9jtqg\" (UID: \"8929ae32-e84a-4a3b-b7ad-5ee3f7575298\") " pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:43:49.307181 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.307018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:43:49.307181 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:49.307133    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:43:49.307181 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:49.307170    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 17:43:49.307295 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:49.307184    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df8c856dc-jqsbv: secret "image-registry-tls" not found
Jul 24 17:43:49.307295 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:49.307221    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls podName:8929ae32-e84a-4a3b-b7ad-5ee3f7575298 nodeName:}" failed. No retries permitted until 2026-07-24 17:44:05.307199646 +0000 UTC m=+67.830558114 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls") pod "dns-default-9jtqg" (UID: "8929ae32-e84a-4a3b-b7ad-5ee3f7575298") : secret "dns-default-metrics-tls" not found
Jul 24 17:43:49.307295 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:49.307234    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls podName:33baa17d-da25-4f28-881f-55035131c237 nodeName:}" failed. No retries permitted until 2026-07-24 17:44:05.307228837 +0000 UTC m=+67.830587297 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls") pod "image-registry-7df8c856dc-jqsbv" (UID: "33baa17d-da25-4f28-881f-55035131c237") : secret "image-registry-tls" not found
Jul 24 17:43:49.387573 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.387544    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:49.710162 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.710125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:49.710162 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.710165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:49.710365 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:49.710266    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 17:43:49.710365 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:49.710316    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls podName:d8d0a6d2-c541-427e-b1e2-48fa4db65735 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:50.710303007 +0000 UTC m=+53.233661463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pfqcd" (UID: "d8d0a6d2-c541-427e-b1e2-48fa4db65735") : secret "insights-runtime-extractor-tls" not found
Jul 24 17:43:49.710365 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:49.710343    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 17:43:49.710468 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:49.710415    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-rbac-proxy-cm podName:d8d0a6d2-c541-427e-b1e2-48fa4db65735 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:50.710402495 +0000 UTC m=+53.233760950 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pfqcd" (UID: "d8d0a6d2-c541-427e-b1e2-48fa4db65735") : configmap "kube-rbac-proxy" not found
Jul 24 17:43:49.764061 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.764026    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:49.865575 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:49.865538    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:50.032014 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:50.031940    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:50.137713 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:50.137679    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:50.350511 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:50.350485    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:50.451131 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:50.451108    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:50.718136 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:50.718052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:50.718136 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:50.718105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:50.718314 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:50.718180    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 17:43:50.718314 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:50.718194    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 17:43:50.718314 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:50.718242    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-rbac-proxy-cm podName:d8d0a6d2-c541-427e-b1e2-48fa4db65735 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:52.718227565 +0000 UTC m=+55.241586021 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pfqcd" (UID: "d8d0a6d2-c541-427e-b1e2-48fa4db65735") : configmap "kube-rbac-proxy" not found
Jul 24 17:43:50.718314 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:50.718255    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls podName:d8d0a6d2-c541-427e-b1e2-48fa4db65735 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:52.718249519 +0000 UTC m=+55.241607975 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pfqcd" (UID: "d8d0a6d2-c541-427e-b1e2-48fa4db65735") : secret "insights-runtime-extractor-tls" not found
Jul 24 17:43:51.061464 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:51.061443    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mn4d8_afc9b6f4-a944-4228-97db-a0c9bbc70a7f/dns-node-resolver/0.log"
Jul 24 17:43:51.656481 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:51.656448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:51.720997 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:51.720968    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:51.864355 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:51.864330    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bnjwn_65c7cb53-638b-4934-9a59-ca69c44a4373/node-ca/0.log"
Jul 24 17:43:52.686482 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:52.686449    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:52.733149 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:52.733120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:52.733149 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:52.733151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:52.733324 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:52.733228    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 17:43:52.733324 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:52.733251    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 17:43:52.733324 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:52.733307    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls podName:d8d0a6d2-c541-427e-b1e2-48fa4db65735 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:56.733293481 +0000 UTC m=+59.256651940 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pfqcd" (UID: "d8d0a6d2-c541-427e-b1e2-48fa4db65735") : secret "insights-runtime-extractor-tls" not found
Jul 24 17:43:52.733324 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:52.733320    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-rbac-proxy-cm podName:d8d0a6d2-c541-427e-b1e2-48fa4db65735 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:56.733313619 +0000 UTC m=+59.256672076 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pfqcd" (UID: "d8d0a6d2-c541-427e-b1e2-48fa4db65735") : configmap "kube-rbac-proxy" not found
Jul 24 17:43:53.059880 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:53.059851    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x95l7_9f1aee3c-1324-4eaa-8201-95c26602d6c1/serve-healthcheck-canary/0.log"
Jul 24 17:43:55.942924 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:55.942890    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:56.039247 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:56.039216    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:56.161448 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:56.161413    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:56.263208 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:56.263129    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:56.372879 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:56.372845    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:56.471177 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:56.471148    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:56.681227 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:56.681197    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:56.758375 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:56.758340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:56.758375 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:56.758374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:43:56.758607 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:56.758467    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 17:43:56.758607 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:56.758497    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 17:43:56.758607 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:56.758552    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-rbac-proxy-cm podName:d8d0a6d2-c541-427e-b1e2-48fa4db65735 nodeName:}" failed. No retries permitted until 2026-07-24 17:44:04.75853247 +0000 UTC m=+67.281890941 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pfqcd" (UID: "d8d0a6d2-c541-427e-b1e2-48fa4db65735") : configmap "kube-rbac-proxy" not found
Jul 24 17:43:56.758607 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:43:56.758578    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls podName:d8d0a6d2-c541-427e-b1e2-48fa4db65735 nodeName:}" failed. No retries permitted until 2026-07-24 17:44:04.758567496 +0000 UTC m=+67.281925963 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pfqcd" (UID: "d8d0a6d2-c541-427e-b1e2-48fa4db65735") : secret "insights-runtime-extractor-tls" not found
Jul 24 17:43:57.472501 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:57.472471    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:58.070871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:58.070835    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:58.124659 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:58.124633    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-46bfx"
Jul 24 17:43:58.263906 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:58.263842    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:58.670582 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:58.670552    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:58.872033 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:58.871998    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:59.266914 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:59.266883    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:59.781266 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:59.781233    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:59.882832 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:59.882797    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:59.937121 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:43:59.937088    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:00.273344 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:00.273310    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:00.466360 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:00.466335    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:00.547509 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:00.547481    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:00.871272 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:00.871194    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:00.980771 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:00.980744    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:01.073280 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:01.073254    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:01.473375 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:01.473345    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:01.873637 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:01.873595    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:02.267529 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:02.267447    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:02.475882 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:02.475852    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:02.576024 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:02.575999    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:02.706307 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:02.706274    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:02.864775 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:02.864702    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:03.444714 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:03.444683    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:03.508932 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:03.508905    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:03.606156 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:03.606125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs\") pod \"network-metrics-daemon-d4gbz\" (UID: \"64e0f517-9969-47f1-8daf-8dfa194ba923\") " pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:44:03.608572 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:03.608554    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 17:44:03.616560 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:44:03.616542    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 24 17:44:03.616621 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:44:03.616611    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs podName:64e0f517-9969-47f1-8daf-8dfa194ba923 nodeName:}" failed. No retries permitted until 2026-07-24 17:45:07.616590519 +0000 UTC m=+130.139948992 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs") pod "network-metrics-daemon-d4gbz" (UID: "64e0f517-9969-47f1-8daf-8dfa194ba923") : secret "metrics-daemon-secret" not found
Jul 24 17:44:03.707169 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:03.707096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret\") pod \"global-pull-secret-syncer-mmrr7\" (UID: \"4c849bf6-e357-4dca-8005-56fbbcb8198a\") " pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:44:03.709610 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:03.709593    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 17:44:03.719866 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:03.719842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c849bf6-e357-4dca-8005-56fbbcb8198a-original-pull-secret\") pod \"global-pull-secret-syncer-mmrr7\" (UID: \"4c849bf6-e357-4dca-8005-56fbbcb8198a\") " pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:44:03.728334 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:03.728310    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:03.807907 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:03.807860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxnc9\" (UniqueName: \"kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9\") pod \"network-check-target-8g49q\" (UID: \"b4d50741-70f0-4e3b-90aa-f2fcaa98201f\") " pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:44:03.810494 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:03.810477    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 17:44:03.821189 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:03.821170    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 17:44:03.831718 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:03.831694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxnc9\" (UniqueName: \"kubernetes.io/projected/b4d50741-70f0-4e3b-90aa-f2fcaa98201f-kube-api-access-vxnc9\") pod \"network-check-target-8g49q\" (UID: \"b4d50741-70f0-4e3b-90aa-f2fcaa98201f\") " pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:44:03.874617 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:03.874593    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9txdv\""
Jul 24 17:44:03.882793 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:03.882765    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:44:03.971128 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:03.971062    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmrr7"
Jul 24 17:44:04.021536 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:04.021512    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8g49q"]
Jul 24 17:44:04.024934 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:44:04.024910    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d50741_70f0_4e3b_90aa_f2fcaa98201f.slice/crio-cd92d8d98e72fc2d9217ab86d8aa7a737e135d9b9c88b150db881d1cf52769f9 WatchSource:0}: Error finding container cd92d8d98e72fc2d9217ab86d8aa7a737e135d9b9c88b150db881d1cf52769f9: Status 404 returned error can't find the container with id cd92d8d98e72fc2d9217ab86d8aa7a737e135d9b9c88b150db881d1cf52769f9
Jul 24 17:44:04.090230 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:04.090202    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mmrr7"]
Jul 24 17:44:04.093221 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:44:04.093188    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c849bf6_e357_4dca_8005_56fbbcb8198a.slice/crio-b32eb2feb2bdd4ab326e4a057b67f9de338ece6f7a34a272165e4862fb051ff9 WatchSource:0}: Error finding container b32eb2feb2bdd4ab326e4a057b67f9de338ece6f7a34a272165e4862fb051ff9: Status 404 returned error can't find the container with id b32eb2feb2bdd4ab326e4a057b67f9de338ece6f7a34a272165e4862fb051ff9
Jul 24 17:44:04.213508 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:04.213473    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mmrr7" event={"ID":"4c849bf6-e357-4dca-8005-56fbbcb8198a","Type":"ContainerStarted","Data":"b32eb2feb2bdd4ab326e4a057b67f9de338ece6f7a34a272165e4862fb051ff9"}
Jul 24 17:44:04.214556 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:04.214535    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8g49q" event={"ID":"b4d50741-70f0-4e3b-90aa-f2fcaa98201f","Type":"ContainerStarted","Data":"cd92d8d98e72fc2d9217ab86d8aa7a737e135d9b9c88b150db881d1cf52769f9"}
Jul 24 17:44:04.279173 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:04.279105    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:04.817884 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:04.817272    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:44:04.817884 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:04.817324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:44:04.817884 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:44:04.817435    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 17:44:04.817884 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:44:04.817504    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls podName:d8d0a6d2-c541-427e-b1e2-48fa4db65735 nodeName:}" failed. No retries permitted until 2026-07-24 17:44:20.817485185 +0000 UTC m=+83.340843656 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pfqcd" (UID: "d8d0a6d2-c541-427e-b1e2-48fa4db65735") : secret "insights-runtime-extractor-tls" not found
Jul 24 17:44:04.818488 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:04.817904    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8d0a6d2-c541-427e-b1e2-48fa4db65735-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:44:04.864565 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:04.864531    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:05.078208 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:05.078089    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:05.320795 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:05.320751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls\") pod \"dns-default-9jtqg\" (UID: \"8929ae32-e84a-4a3b-b7ad-5ee3f7575298\") " pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:44:05.320983 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:05.320824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:44:05.323988 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:05.323928    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8929ae32-e84a-4a3b-b7ad-5ee3f7575298-metrics-tls\") pod \"dns-default-9jtqg\" (UID: \"8929ae32-e84a-4a3b-b7ad-5ee3f7575298\") " pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:44:05.324120 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:05.324093    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls\") pod \"image-registry-7df8c856dc-jqsbv\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") " pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:44:05.466854 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:05.466756    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:05.504520 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:05.504484    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5kpm4\""
Jul 24 17:44:05.511389 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:05.511357    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vh2n6\""
Jul 24 17:44:05.512523 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:05.512370    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:44:05.519429 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:05.519408    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:44:05.665610 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:05.665577    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:05.670989 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:05.670917    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9jtqg"]
Jul 24 17:44:05.675102 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:44:05.675072    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8929ae32_e84a_4a3b_b7ad_5ee3f7575298.slice/crio-cde687b5757f302a9b36986c2d73cf94f2685b0215b28b4c9a09dc83103eeee2 WatchSource:0}: Error finding container cde687b5757f302a9b36986c2d73cf94f2685b0215b28b4c9a09dc83103eeee2: Status 404 returned error can't find the container with id cde687b5757f302a9b36986c2d73cf94f2685b0215b28b4c9a09dc83103eeee2
Jul 24 17:44:05.693087 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:05.693046    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7df8c856dc-jqsbv"]
Jul 24 17:44:05.696199 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:44:05.696169    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33baa17d_da25_4f28_881f_55035131c237.slice/crio-b265c8bdbb2dd0f3b42fa367a7efc808ac16890efc9d2ce3d480da1fba1f5b73 WatchSource:0}: Error finding container b265c8bdbb2dd0f3b42fa367a7efc808ac16890efc9d2ce3d480da1fba1f5b73: Status 404 returned error can't find the container with id b265c8bdbb2dd0f3b42fa367a7efc808ac16890efc9d2ce3d480da1fba1f5b73
Jul 24 17:44:06.071244 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:06.071192    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:06.222066 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:06.221773    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv" event={"ID":"33baa17d-da25-4f28-881f-55035131c237","Type":"ContainerStarted","Data":"ef5b980dff801ddeaa6590b3f0295d1177a49a8bd211d155a7dc248f13f9b135"}
Jul 24 17:44:06.222066 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:06.221846    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv" event={"ID":"33baa17d-da25-4f28-881f-55035131c237","Type":"ContainerStarted","Data":"b265c8bdbb2dd0f3b42fa367a7efc808ac16890efc9d2ce3d480da1fba1f5b73"}
Jul 24 17:44:06.222066 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:06.221920    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:44:06.223222 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:06.223191    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9jtqg" event={"ID":"8929ae32-e84a-4a3b-b7ad-5ee3f7575298","Type":"ContainerStarted","Data":"cde687b5757f302a9b36986c2d73cf94f2685b0215b28b4c9a09dc83103eeee2"}
Jul 24 17:44:06.244420 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:06.244375    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv" podStartSLOduration=54.244361759 podStartE2EDuration="54.244361759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:06.243583012 +0000 UTC m=+68.766941490" watchObservedRunningTime="2026-07-24 17:44:06.244361759 +0000 UTC m=+68.767720238"
Jul 24 17:44:06.469862 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:06.469721    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:06.665234 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:06.665200    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:07.195285 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:07.195252    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:07.296386 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:07.296354    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:07.349116 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:07.349078    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:07.489616 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:07.489552    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:07.854824 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:07.854775    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:08.273960 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:08.273879    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:08.327568 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:08.327532    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:08.487161 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:08.487120    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:08.575684 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:08.575655    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:08.871636 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:08.871573    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:08.986325 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:08.986291    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:09.099126 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:09.099091    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:09.206053 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:09.205970    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:09.354295 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:09.354265    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:09.455285 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:09.455244    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:10.235535 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:10.235499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mmrr7" event={"ID":"4c849bf6-e357-4dca-8005-56fbbcb8198a","Type":"ContainerStarted","Data":"727a4759f6052e2f57b15dcbabb062e5c01d417a7c1497e4d9ed71cac3e149b2"}
Jul 24 17:44:10.236960 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:10.236930    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8g49q" event={"ID":"b4d50741-70f0-4e3b-90aa-f2fcaa98201f","Type":"ContainerStarted","Data":"bb2f94a57f1ce983ee41ea021b895f501918bb7ed52b6dc85ffe74338990596f"}
Jul 24 17:44:10.237097 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:10.237017    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:44:10.238493 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:10.238473    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9jtqg" event={"ID":"8929ae32-e84a-4a3b-b7ad-5ee3f7575298","Type":"ContainerStarted","Data":"7437ea50c45063b73c86d3682a6a3b5ca67f6a813e9168bd71925953e7b8124b"}
Jul 24 17:44:10.238560 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:10.238508    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9jtqg" event={"ID":"8929ae32-e84a-4a3b-b7ad-5ee3f7575298","Type":"ContainerStarted","Data":"7ed8a3a19d919827ffa35e1cf0e0c4630037a60144936b7cfab052a03ab5f333"}
Jul 24 17:44:10.238560 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:10.238533    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:44:10.268490 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:10.268447    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mmrr7" podStartSLOduration=67.009270981 podStartE2EDuration="1m12.268430462s" totalImagesPullingTime="5.259159481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:10.252482196 +0000 UTC m=+72.775840673" watchObservedRunningTime="2026-07-24 17:44:10.268430462 +0000 UTC m=+72.791788944"
Jul 24 17:44:10.268995 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:10.268966    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8g49q" podStartSLOduration=66.956082526 podStartE2EDuration="1m12.268956681s" totalImagesPullingTime="5.312874155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:10.268008749 +0000 UTC m=+72.791367228" watchObservedRunningTime="2026-07-24 17:44:10.268956681 +0000 UTC m=+72.792315158"
Jul 24 17:44:10.286916 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:10.286873    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9jtqg" podStartSLOduration=34.619378637 podStartE2EDuration="38.286862204s" totalImagesPullingTime="3.667483567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:10.28653909 +0000 UTC m=+72.809897581" watchObservedRunningTime="2026-07-24 17:44:10.286862204 +0000 UTC m=+72.810220682"
Jul 24 17:44:10.492535 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:10.492441    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:10.555689 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:10.555665    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:12.188335 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:12.188305    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:13.874486 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:13.874449    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:13.930731 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:13.930701    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:15.764338 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:15.764307    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-kx42n"]
Jul 24 17:44:15.766510 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:15.766491    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-kx42n"
Jul 24 17:44:15.769322 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:15.769299    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 24 17:44:15.769428 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:15.769328    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-knjsj\""
Jul 24 17:44:15.769428 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:15.769413    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 24 17:44:15.784417 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:15.784389    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-kx42n"]
Jul 24 17:44:15.790922 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:15.790903    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7df8c856dc-jqsbv"]
Jul 24 17:44:15.800702 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:15.800679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp2pl\" (UniqueName: \"kubernetes.io/projected/0a303a45-e8f0-4b53-940b-d22569b73afb-kube-api-access-mp2pl\") pod \"downloads-7c97dc4555-kx42n\" (UID: \"0a303a45-e8f0-4b53-940b-d22569b73afb\") " pod="openshift-console/downloads-7c97dc4555-kx42n"
Jul 24 17:44:15.895331 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:15.895303    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-657dd9f654-2fvxt"]
Jul 24 17:44:15.896971 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:15.896956    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:15.901589 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:15.901546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp2pl\" (UniqueName: \"kubernetes.io/projected/0a303a45-e8f0-4b53-940b-d22569b73afb-kube-api-access-mp2pl\") pod \"downloads-7c97dc4555-kx42n\" (UID: \"0a303a45-e8f0-4b53-940b-d22569b73afb\") " pod="openshift-console/downloads-7c97dc4555-kx42n"
Jul 24 17:44:15.921596 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:15.921573    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-657dd9f654-2fvxt"]
Jul 24 17:44:15.926485 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:15.926461    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp2pl\" (UniqueName: \"kubernetes.io/projected/0a303a45-e8f0-4b53-940b-d22569b73afb-kube-api-access-mp2pl\") pod \"downloads-7c97dc4555-kx42n\" (UID: \"0a303a45-e8f0-4b53-940b-d22569b73afb\") " pod="openshift-console/downloads-7c97dc4555-kx42n"
Jul 24 17:44:15.936088 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:15.936063    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:16.001975 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.001947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3692f369-8fc2-4b2f-b064-034d5f50957c-image-registry-private-configuration\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.002102 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.001980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3692f369-8fc2-4b2f-b064-034d5f50957c-ca-trust-extracted\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.002102 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.002001    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3692f369-8fc2-4b2f-b064-034d5f50957c-registry-tls\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.002102 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.002016    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3692f369-8fc2-4b2f-b064-034d5f50957c-bound-sa-token\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.002102 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.002061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3692f369-8fc2-4b2f-b064-034d5f50957c-installation-pull-secrets\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.002102 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.002089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3692f369-8fc2-4b2f-b064-034d5f50957c-trusted-ca\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.002286 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.002129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3692f369-8fc2-4b2f-b064-034d5f50957c-registry-certificates\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.002286 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.002155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvbgx\" (UniqueName: \"kubernetes.io/projected/3692f369-8fc2-4b2f-b064-034d5f50957c-kube-api-access-rvbgx\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.010014 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.009988    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:16.074887 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.074860    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-kx42n"
Jul 24 17:44:16.102516 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.102492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3692f369-8fc2-4b2f-b064-034d5f50957c-registry-certificates\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.102619 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.102539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvbgx\" (UniqueName: \"kubernetes.io/projected/3692f369-8fc2-4b2f-b064-034d5f50957c-kube-api-access-rvbgx\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.102619 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.102582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3692f369-8fc2-4b2f-b064-034d5f50957c-image-registry-private-configuration\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.102619 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.102603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3692f369-8fc2-4b2f-b064-034d5f50957c-ca-trust-extracted\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.102755 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.102624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3692f369-8fc2-4b2f-b064-034d5f50957c-registry-tls\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.102755 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.102645    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3692f369-8fc2-4b2f-b064-034d5f50957c-bound-sa-token\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.102755 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.102706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3692f369-8fc2-4b2f-b064-034d5f50957c-installation-pull-secrets\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.102755 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.102738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3692f369-8fc2-4b2f-b064-034d5f50957c-trusted-ca\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.103022 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.102977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3692f369-8fc2-4b2f-b064-034d5f50957c-ca-trust-extracted\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.103307 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.103284    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3692f369-8fc2-4b2f-b064-034d5f50957c-registry-certificates\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.104418 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.104391    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3692f369-8fc2-4b2f-b064-034d5f50957c-trusted-ca\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.105150 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.105126    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3692f369-8fc2-4b2f-b064-034d5f50957c-image-registry-private-configuration\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.105479 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.105460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3692f369-8fc2-4b2f-b064-034d5f50957c-registry-tls\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.105558 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.105473    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3692f369-8fc2-4b2f-b064-034d5f50957c-installation-pull-secrets\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.111151 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.111128    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3692f369-8fc2-4b2f-b064-034d5f50957c-bound-sa-token\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.111330 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.111232    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvbgx\" (UniqueName: \"kubernetes.io/projected/3692f369-8fc2-4b2f-b064-034d5f50957c-kube-api-access-rvbgx\") pod \"image-registry-657dd9f654-2fvxt\" (UID: \"3692f369-8fc2-4b2f-b064-034d5f50957c\") " pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.126698 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.126496    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:16.197257 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.197223    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-kx42n"]
Jul 24 17:44:16.201041 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:44:16.201012    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a303a45_e8f0_4b53_940b_d22569b73afb.slice/crio-c2346f8712380eab47dae9d2dde626200dc0f006852800429065101e2c5815b7 WatchSource:0}: Error finding container c2346f8712380eab47dae9d2dde626200dc0f006852800429065101e2c5815b7: Status 404 returned error can't find the container with id c2346f8712380eab47dae9d2dde626200dc0f006852800429065101e2c5815b7
Jul 24 17:44:16.201969 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.201947    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:16.205937 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.205658    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:16.260347 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.260307    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-kx42n" event={"ID":"0a303a45-e8f0-4b53-940b-d22569b73afb","Type":"ContainerStarted","Data":"c2346f8712380eab47dae9d2dde626200dc0f006852800429065101e2c5815b7"}
Jul 24 17:44:16.329600 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.329580    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-657dd9f654-2fvxt"]
Jul 24 17:44:16.331831 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:44:16.331805    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3692f369_8fc2_4b2f_b064_034d5f50957c.slice/crio-ec2464f2ad9943532f434d4fdb9d2432a3a5f918638ba18e5787199096d6a8a6 WatchSource:0}: Error finding container ec2464f2ad9943532f434d4fdb9d2432a3a5f918638ba18e5787199096d6a8a6: Status 404 returned error can't find the container with id ec2464f2ad9943532f434d4fdb9d2432a3a5f918638ba18e5787199096d6a8a6
Jul 24 17:44:16.729232 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.729162    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:16.843285 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:16.843248    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:17.267214 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:17.267177    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657dd9f654-2fvxt" event={"ID":"3692f369-8fc2-4b2f-b064-034d5f50957c","Type":"ContainerStarted","Data":"7d02beb47b973ca36dd3df79e858180b9bba4042d8738af4e359053af58fdd7d"}
Jul 24 17:44:17.267398 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:17.267235    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657dd9f654-2fvxt" event={"ID":"3692f369-8fc2-4b2f-b064-034d5f50957c","Type":"ContainerStarted","Data":"ec2464f2ad9943532f434d4fdb9d2432a3a5f918638ba18e5787199096d6a8a6"}
Jul 24 17:44:17.267526 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:17.267478    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:17.288332 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:17.288283    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-657dd9f654-2fvxt" podStartSLOduration=2.2882622440000002 podStartE2EDuration="2.288262244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:17.288153983 +0000 UTC m=+79.811512461" watchObservedRunningTime="2026-07-24 17:44:17.288262244 +0000 UTC m=+79.811620706"
Jul 24 17:44:18.390218 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.390179    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:18.430361 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.430328    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-k7f9v"]
Jul 24 17:44:18.432719 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.432693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-k7f9v"
Jul 24 17:44:18.436462 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.436439    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 17:44:18.436573 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.436476    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 24 17:44:18.436573 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.436504    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 17:44:18.436573 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.436439    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 17:44:18.436728 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.436440    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 24 17:44:18.436728 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.436712    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5nvds\""
Jul 24 17:44:18.444808 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.444770    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-k7f9v"]
Jul 24 17:44:18.468325 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.468282    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:18.521176 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.521144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/532a4ad5-58b1-48b3-b07e-b033f4a7bde4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-k7f9v\" (UID: \"532a4ad5-58b1-48b3-b07e-b033f4a7bde4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k7f9v"
Jul 24 17:44:18.521338 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.521204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/532a4ad5-58b1-48b3-b07e-b033f4a7bde4-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-k7f9v\" (UID: \"532a4ad5-58b1-48b3-b07e-b033f4a7bde4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k7f9v"
Jul 24 17:44:18.521338 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.521234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/532a4ad5-58b1-48b3-b07e-b033f4a7bde4-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-k7f9v\" (UID: \"532a4ad5-58b1-48b3-b07e-b033f4a7bde4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k7f9v"
Jul 24 17:44:18.521338 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.521258    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qs99\" (UniqueName: \"kubernetes.io/projected/532a4ad5-58b1-48b3-b07e-b033f4a7bde4-kube-api-access-5qs99\") pod \"prometheus-operator-84d69f794d-k7f9v\" (UID: \"532a4ad5-58b1-48b3-b07e-b033f4a7bde4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k7f9v"
Jul 24 17:44:18.622123 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.622081    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/532a4ad5-58b1-48b3-b07e-b033f4a7bde4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-k7f9v\" (UID: \"532a4ad5-58b1-48b3-b07e-b033f4a7bde4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k7f9v"
Jul 24 17:44:18.622292 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.622160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/532a4ad5-58b1-48b3-b07e-b033f4a7bde4-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-k7f9v\" (UID: \"532a4ad5-58b1-48b3-b07e-b033f4a7bde4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k7f9v"
Jul 24 17:44:18.622292 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.622190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/532a4ad5-58b1-48b3-b07e-b033f4a7bde4-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-k7f9v\" (UID: \"532a4ad5-58b1-48b3-b07e-b033f4a7bde4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k7f9v"
Jul 24 17:44:18.622292 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.622216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qs99\" (UniqueName: \"kubernetes.io/projected/532a4ad5-58b1-48b3-b07e-b033f4a7bde4-kube-api-access-5qs99\") pod \"prometheus-operator-84d69f794d-k7f9v\" (UID: \"532a4ad5-58b1-48b3-b07e-b033f4a7bde4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k7f9v"
Jul 24 17:44:18.622520 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:44:18.622306    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 24 17:44:18.622520 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:44:18.622385    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/532a4ad5-58b1-48b3-b07e-b033f4a7bde4-prometheus-operator-tls podName:532a4ad5-58b1-48b3-b07e-b033f4a7bde4 nodeName:}" failed. No retries permitted until 2026-07-24 17:44:19.122366905 +0000 UTC m=+81.645725362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/532a4ad5-58b1-48b3-b07e-b033f4a7bde4-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-k7f9v" (UID: "532a4ad5-58b1-48b3-b07e-b033f4a7bde4") : secret "prometheus-operator-tls" not found
Jul 24 17:44:18.623127 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.623103    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/532a4ad5-58b1-48b3-b07e-b033f4a7bde4-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-k7f9v\" (UID: \"532a4ad5-58b1-48b3-b07e-b033f4a7bde4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k7f9v"
Jul 24 17:44:18.624753 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.624728    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/532a4ad5-58b1-48b3-b07e-b033f4a7bde4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-k7f9v\" (UID: \"532a4ad5-58b1-48b3-b07e-b033f4a7bde4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k7f9v"
Jul 24 17:44:18.632810 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.632768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qs99\" (UniqueName: \"kubernetes.io/projected/532a4ad5-58b1-48b3-b07e-b033f4a7bde4-kube-api-access-5qs99\") pod \"prometheus-operator-84d69f794d-k7f9v\" (UID: \"532a4ad5-58b1-48b3-b07e-b033f4a7bde4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k7f9v"
Jul 24 17:44:18.825566 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:18.825527    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:19.126154 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:19.126064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/532a4ad5-58b1-48b3-b07e-b033f4a7bde4-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-k7f9v\" (UID: \"532a4ad5-58b1-48b3-b07e-b033f4a7bde4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k7f9v"
Jul 24 17:44:19.128717 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:19.128604    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/532a4ad5-58b1-48b3-b07e-b033f4a7bde4-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-k7f9v\" (UID: \"532a4ad5-58b1-48b3-b07e-b033f4a7bde4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k7f9v"
Jul 24 17:44:19.345328 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:19.345288    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-k7f9v"
Jul 24 17:44:19.482324 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:19.482287    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-k7f9v"]
Jul 24 17:44:19.485745 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:44:19.485716    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532a4ad5_58b1_48b3_b07e_b033f4a7bde4.slice/crio-feafbf013a4260d691069b22dca6869b20d06b0d9c4acf54bd6feb4d372ba3f2 WatchSource:0}: Error finding container feafbf013a4260d691069b22dca6869b20d06b0d9c4acf54bd6feb4d372ba3f2: Status 404 returned error can't find the container with id feafbf013a4260d691069b22dca6869b20d06b0d9c4acf54bd6feb4d372ba3f2
Jul 24 17:44:20.243956 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:20.243921    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9jtqg"
Jul 24 17:44:20.283580 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:20.283499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-k7f9v" event={"ID":"532a4ad5-58b1-48b3-b07e-b033f4a7bde4","Type":"ContainerStarted","Data":"feafbf013a4260d691069b22dca6869b20d06b0d9c4acf54bd6feb4d372ba3f2"}
Jul 24 17:44:20.840887 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:20.840854    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:44:20.843258 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:20.843232    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8d0a6d2-c541-427e-b1e2-48fa4db65735-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfqcd\" (UID: \"d8d0a6d2-c541-427e-b1e2-48fa4db65735\") " pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:44:21.118755 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:21.118666    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cg7vk\""
Jul 24 17:44:21.126657 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:21.126592    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pfqcd"
Jul 24 17:44:21.169203 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:21.169164    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:21.274847 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:21.274813    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pfqcd"]
Jul 24 17:44:21.279774 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:44:21.279745    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d0a6d2_c541_427e_b1e2_48fa4db65735.slice/crio-81e630783165bbf5265f59fe65e4e5d8ab6d2f491d807229ef48bc53cab56128 WatchSource:0}: Error finding container 81e630783165bbf5265f59fe65e4e5d8ab6d2f491d807229ef48bc53cab56128: Status 404 returned error can't find the container with id 81e630783165bbf5265f59fe65e4e5d8ab6d2f491d807229ef48bc53cab56128
Jul 24 17:44:21.288915 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:21.288885    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-k7f9v" event={"ID":"532a4ad5-58b1-48b3-b07e-b033f4a7bde4","Type":"ContainerStarted","Data":"949abe9601f37442d8811892a475ea5d0198fdf78d3bb8e13d87a84299f21813"}
Jul 24 17:44:21.289015 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:21.288951    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-k7f9v" event={"ID":"532a4ad5-58b1-48b3-b07e-b033f4a7bde4","Type":"ContainerStarted","Data":"f847e62dd808a810fdcea777a5b8a1764692b2738d73bd1dc46c67e14d106ace"}
Jul 24 17:44:21.290077 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:21.290054    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pfqcd" event={"ID":"d8d0a6d2-c541-427e-b1e2-48fa4db65735","Type":"ContainerStarted","Data":"81e630783165bbf5265f59fe65e4e5d8ab6d2f491d807229ef48bc53cab56128"}
Jul 24 17:44:21.308391 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:21.308342    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-k7f9v" podStartSLOduration=1.902714411 podStartE2EDuration="3.308324613s" totalImagesPullingTime="1.405610202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:21.307180168 +0000 UTC m=+83.830538649" watchObservedRunningTime="2026-07-24 17:44:21.308324613 +0000 UTC m=+83.831683096"
Jul 24 17:44:21.698282 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:21.698250    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:21.768650 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:21.768620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:22.540907 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:22.540870    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:22.820045 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:22.820019    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:22.830013 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:22.829988    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pfvzx"]
Jul 24 17:44:22.833330 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:22.833304    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:22.836608 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:22.836440    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 17:44:22.836608 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:22.836483    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 17:44:22.836608 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:22.836517    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 17:44:22.836608 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:22.836584    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mfdvq\""
Jul 24 17:44:22.909583 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:22.909550    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:22.959104 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:22.959051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/556b6001-3830-478a-afd7-9cd9bc3a9926-node-exporter-wtmp\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:22.959277 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:22.959136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/556b6001-3830-478a-afd7-9cd9bc3a9926-node-exporter-tls\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:22.959277 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:22.959207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/556b6001-3830-478a-afd7-9cd9bc3a9926-metrics-client-ca\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:22.959277 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:22.959231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/556b6001-3830-478a-afd7-9cd9bc3a9926-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:22.959445 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:22.959290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/556b6001-3830-478a-afd7-9cd9bc3a9926-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:22.959445 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:22.959322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcdlh\" (UniqueName: \"kubernetes.io/projected/556b6001-3830-478a-afd7-9cd9bc3a9926-kube-api-access-lcdlh\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:22.959445 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:22.959349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/556b6001-3830-478a-afd7-9cd9bc3a9926-node-exporter-textfile\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:22.959445 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:22.959386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/556b6001-3830-478a-afd7-9cd9bc3a9926-sys\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:22.959445 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:22.959426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/556b6001-3830-478a-afd7-9cd9bc3a9926-root\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.014793 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.014755    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:23.060185 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.060156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/556b6001-3830-478a-afd7-9cd9bc3a9926-node-exporter-tls\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.060348 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.060198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/556b6001-3830-478a-afd7-9cd9bc3a9926-metrics-client-ca\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.060348 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.060226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/556b6001-3830-478a-afd7-9cd9bc3a9926-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.060348 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.060261    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/556b6001-3830-478a-afd7-9cd9bc3a9926-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.060348 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.060288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcdlh\" (UniqueName: \"kubernetes.io/projected/556b6001-3830-478a-afd7-9cd9bc3a9926-kube-api-access-lcdlh\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.060348 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.060312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/556b6001-3830-478a-afd7-9cd9bc3a9926-node-exporter-textfile\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.060348 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:44:23.060321    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 24 17:44:23.060348 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.060336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/556b6001-3830-478a-afd7-9cd9bc3a9926-sys\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.060709 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:44:23.060391    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/556b6001-3830-478a-afd7-9cd9bc3a9926-node-exporter-tls podName:556b6001-3830-478a-afd7-9cd9bc3a9926 nodeName:}" failed. No retries permitted until 2026-07-24 17:44:23.560369399 +0000 UTC m=+86.083727875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/556b6001-3830-478a-afd7-9cd9bc3a9926-node-exporter-tls") pod "node-exporter-pfvzx" (UID: "556b6001-3830-478a-afd7-9cd9bc3a9926") : secret "node-exporter-tls" not found
Jul 24 17:44:23.060709 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.060397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/556b6001-3830-478a-afd7-9cd9bc3a9926-sys\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.060709 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.060520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/556b6001-3830-478a-afd7-9cd9bc3a9926-root\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.060709 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.060591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/556b6001-3830-478a-afd7-9cd9bc3a9926-node-exporter-wtmp\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.060709 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.060653    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/556b6001-3830-478a-afd7-9cd9bc3a9926-root\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.061010 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.060759    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/556b6001-3830-478a-afd7-9cd9bc3a9926-node-exporter-wtmp\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.061010 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.060857    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/556b6001-3830-478a-afd7-9cd9bc3a9926-metrics-client-ca\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.061010 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.060901    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/556b6001-3830-478a-afd7-9cd9bc3a9926-node-exporter-textfile\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.061176 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.061152    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/556b6001-3830-478a-afd7-9cd9bc3a9926-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.063297 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.063254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/556b6001-3830-478a-afd7-9cd9bc3a9926-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.073370 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.073295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcdlh\" (UniqueName: \"kubernetes.io/projected/556b6001-3830-478a-afd7-9cd9bc3a9926-kube-api-access-lcdlh\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.108474 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.108442    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:23.209146 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.209113    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:23.289975 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.289941    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:23.468194 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.468080    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:23.565121 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.565076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/556b6001-3830-478a-afd7-9cd9bc3a9926-node-exporter-tls\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.567915 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.567891    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/556b6001-3830-478a-afd7-9cd9bc3a9926-node-exporter-tls\") pod \"node-exporter-pfvzx\" (UID: \"556b6001-3830-478a-afd7-9cd9bc3a9926\") " pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.570414 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.570361    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:23.747171 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.746418    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pfvzx"
Jul 24 17:44:23.761219 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:44:23.761188    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556b6001_3830_478a_afd7_9cd9bc3a9926.slice/crio-0028ee39b5aaa23c5a5a40a288eb42dfe831e325f07f8fbdea4053fc6f4fc381 WatchSource:0}: Error finding container 0028ee39b5aaa23c5a5a40a288eb42dfe831e325f07f8fbdea4053fc6f4fc381: Status 404 returned error can't find the container with id 0028ee39b5aaa23c5a5a40a288eb42dfe831e325f07f8fbdea4053fc6f4fc381
Jul 24 17:44:23.811441 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.811407    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:23.900379 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:23.900310    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:24.017744 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:24.017667    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:24.303953 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:24.303916    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfvzx" event={"ID":"556b6001-3830-478a-afd7-9cd9bc3a9926","Type":"ContainerStarted","Data":"0028ee39b5aaa23c5a5a40a288eb42dfe831e325f07f8fbdea4053fc6f4fc381"}
Jul 24 17:44:25.698749 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.698594    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:25.773836 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.773775    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:25.778113 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.778085    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7bc9895498-jf4x5"]
Jul 24 17:44:25.798820 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.798738    2578 patch_prober.go:28] interesting pod/image-registry-7df8c856dc-jqsbv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 24 17:44:25.798991 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.798842    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv" podUID="33baa17d-da25-4f28-881f-55035131c237" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 17:44:25.807893 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.807797    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bc9895498-jf4x5"]
Jul 24 17:44:25.808035 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.807912    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.811421 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.810655    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-n7bgg\""
Jul 24 17:44:25.811421 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.810700    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 24 17:44:25.811421 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.810936    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-kjrcn1h9a8b\""
Jul 24 17:44:25.811421 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.810949    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 24 17:44:25.811421 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.810968    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 24 17:44:25.811421 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.811070    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 24 17:44:25.811421 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.811183    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 24 17:44:25.884297 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.884265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.884466 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.884306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-metrics-client-ca\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.884466 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.884335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-secret-grpc-tls\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.884466 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.884360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.884593 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.884505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.884593 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.884544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx2d6\" (UniqueName: \"kubernetes.io/projected/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-kube-api-access-lx2d6\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.884593 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.884580    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.884703 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.884672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-secret-thanos-querier-tls\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.985309 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.985222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.985309 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.985269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-metrics-client-ca\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.985309 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.985293    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-secret-grpc-tls\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.985561 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.985320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.985561 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.985380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.985561 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.985404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx2d6\" (UniqueName: \"kubernetes.io/projected/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-kube-api-access-lx2d6\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.985561 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.985438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.985561 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.985539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-secret-thanos-querier-tls\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.986480 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.986452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-metrics-client-ca\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.988382 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.988356    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.988875 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.988833    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.989491 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.989443    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.990131 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.990101    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-secret-grpc-tls\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.990568 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.990525    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-secret-thanos-querier-tls\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.990713 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.990683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:25.994331 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:25.994311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx2d6\" (UniqueName: \"kubernetes.io/projected/419c9f71-0a9d-4a42-b11d-e3eebaa895a1-kube-api-access-lx2d6\") pod \"thanos-querier-7bc9895498-jf4x5\" (UID: \"419c9f71-0a9d-4a42-b11d-e3eebaa895a1\") " pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:26.120319 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:26.120284    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:26.285152 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:26.285071    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:26.381521 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:26.381488    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:26.563607 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:26.563572    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:27.104839 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.104806    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"]
Jul 24 17:44:27.107885 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.107863    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.110862 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.110842    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 24 17:44:27.111950 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.111930    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cskd44463bg6u\""
Jul 24 17:44:27.111950 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.111943    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-77lms\""
Jul 24 17:44:27.112090 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.111965    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 24 17:44:27.112090 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.112004    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 24 17:44:27.112090 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.111930    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 24 17:44:27.118944 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.118921    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"]
Jul 24 17:44:27.195742 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.195710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d8rr\" (UniqueName: \"kubernetes.io/projected/2660087a-bcaa-41c3-b3e3-12070c59fe76-kube-api-access-6d8rr\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.195917 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.195751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2660087a-bcaa-41c3-b3e3-12070c59fe76-secret-metrics-server-tls\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.195917 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.195808    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2660087a-bcaa-41c3-b3e3-12070c59fe76-secret-metrics-server-client-certs\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.195917 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.195832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2660087a-bcaa-41c3-b3e3-12070c59fe76-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.195917 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.195862    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2660087a-bcaa-41c3-b3e3-12070c59fe76-client-ca-bundle\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.196089 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.196011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2660087a-bcaa-41c3-b3e3-12070c59fe76-audit-log\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.196089 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.196055    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2660087a-bcaa-41c3-b3e3-12070c59fe76-metrics-server-audit-profiles\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.296530 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.296497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2660087a-bcaa-41c3-b3e3-12070c59fe76-secret-metrics-server-client-certs\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.296699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.296585    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2660087a-bcaa-41c3-b3e3-12070c59fe76-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.296699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.296614    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:27.296699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.296629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2660087a-bcaa-41c3-b3e3-12070c59fe76-client-ca-bundle\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.296886 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.296708    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2660087a-bcaa-41c3-b3e3-12070c59fe76-audit-log\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.296886 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.296749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2660087a-bcaa-41c3-b3e3-12070c59fe76-metrics-server-audit-profiles\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.296983 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.296895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d8rr\" (UniqueName: \"kubernetes.io/projected/2660087a-bcaa-41c3-b3e3-12070c59fe76-kube-api-access-6d8rr\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.296983 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.296929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2660087a-bcaa-41c3-b3e3-12070c59fe76-secret-metrics-server-tls\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.297090 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.297076    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2660087a-bcaa-41c3-b3e3-12070c59fe76-audit-log\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.297430 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.297384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2660087a-bcaa-41c3-b3e3-12070c59fe76-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.297812 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.297774    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2660087a-bcaa-41c3-b3e3-12070c59fe76-metrics-server-audit-profiles\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.299368 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.299344    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2660087a-bcaa-41c3-b3e3-12070c59fe76-secret-metrics-server-client-certs\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.299560 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.299538    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2660087a-bcaa-41c3-b3e3-12070c59fe76-client-ca-bundle\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.299664 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.299575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2660087a-bcaa-41c3-b3e3-12070c59fe76-secret-metrics-server-tls\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.304943 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.304920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d8rr\" (UniqueName: \"kubernetes.io/projected/2660087a-bcaa-41c3-b3e3-12070c59fe76-kube-api-access-6d8rr\") pod \"metrics-server-585dbd8bc9-5lsgl\" (UID: \"2660087a-bcaa-41c3-b3e3-12070c59fe76\") " pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.397903 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.397812    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:27.419705 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.419677    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:27.567760 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.567726    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:27.584776 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.584742    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-npsd7"]
Jul 24 17:44:27.587981 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.587958    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-npsd7"
Jul 24 17:44:27.590649 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.590627    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6rrth\""
Jul 24 17:44:27.590649 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.590637    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 24 17:44:27.595102 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.595049    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-npsd7"]
Jul 24 17:44:27.670664 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.670560    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:27.699875 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.699838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a2015b61-085f-4832-b797-139af3e5d12b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-npsd7\" (UID: \"a2015b61-085f-4832-b797-139af3e5d12b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-npsd7"
Jul 24 17:44:27.700038 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.699904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv7lq\" (UniqueName: \"kubernetes.io/projected/a2015b61-085f-4832-b797-139af3e5d12b-kube-api-access-mv7lq\") pod \"monitoring-plugin-68b7fc9574-npsd7\" (UID: \"a2015b61-085f-4832-b797-139af3e5d12b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-npsd7"
Jul 24 17:44:27.800618 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.800580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a2015b61-085f-4832-b797-139af3e5d12b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-npsd7\" (UID: \"a2015b61-085f-4832-b797-139af3e5d12b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-npsd7"
Jul 24 17:44:27.800808 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.800649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv7lq\" (UniqueName: \"kubernetes.io/projected/a2015b61-085f-4832-b797-139af3e5d12b-kube-api-access-mv7lq\") pod \"monitoring-plugin-68b7fc9574-npsd7\" (UID: \"a2015b61-085f-4832-b797-139af3e5d12b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-npsd7"
Jul 24 17:44:27.800808 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:44:27.800747    2578 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 24 17:44:27.800913 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:44:27.800835    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2015b61-085f-4832-b797-139af3e5d12b-monitoring-plugin-cert podName:a2015b61-085f-4832-b797-139af3e5d12b nodeName:}" failed. No retries permitted until 2026-07-24 17:44:28.300816202 +0000 UTC m=+90.824174679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a2015b61-085f-4832-b797-139af3e5d12b-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-npsd7" (UID: "a2015b61-085f-4832-b797-139af3e5d12b") : secret "monitoring-plugin-cert" not found
Jul 24 17:44:27.810525 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:27.810501    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv7lq\" (UniqueName: \"kubernetes.io/projected/a2015b61-085f-4832-b797-139af3e5d12b-kube-api-access-mv7lq\") pod \"monitoring-plugin-68b7fc9574-npsd7\" (UID: \"a2015b61-085f-4832-b797-139af3e5d12b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-npsd7"
Jul 24 17:44:28.048241 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.048209    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-cd85757d-vlm7b"]
Jul 24 17:44:28.051574 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.051549    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.055387 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.055366    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 24 17:44:28.055526 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.055372    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 24 17:44:28.055526 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.055374    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-dtm25\""
Jul 24 17:44:28.055828 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.055813    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 24 17:44:28.056133 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.056114    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 24 17:44:28.056665 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.056648    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 24 17:44:28.061916 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.061894    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 24 17:44:28.070996 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.070969    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-cd85757d-vlm7b"]
Jul 24 17:44:28.103578 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.103543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/388ad4ed-8180-4947-b4de-b09e487fc128-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.103770 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.103595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/388ad4ed-8180-4947-b4de-b09e487fc128-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.103770 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.103655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/388ad4ed-8180-4947-b4de-b09e487fc128-serving-certs-ca-bundle\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.103770 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.103689    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flltq\" (UniqueName: \"kubernetes.io/projected/388ad4ed-8180-4947-b4de-b09e487fc128-kube-api-access-flltq\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.103969 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.103800    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/388ad4ed-8180-4947-b4de-b09e487fc128-telemeter-client-tls\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.103969 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.103838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/388ad4ed-8180-4947-b4de-b09e487fc128-secret-telemeter-client\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.103969 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.103910    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/388ad4ed-8180-4947-b4de-b09e487fc128-federate-client-tls\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.103969 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.103964    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/388ad4ed-8180-4947-b4de-b09e487fc128-metrics-client-ca\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.205000 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.204965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/388ad4ed-8180-4947-b4de-b09e487fc128-telemeter-client-tls\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.205474 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.205002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/388ad4ed-8180-4947-b4de-b09e487fc128-secret-telemeter-client\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.205474 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.205063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/388ad4ed-8180-4947-b4de-b09e487fc128-federate-client-tls\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.205474 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.205096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/388ad4ed-8180-4947-b4de-b09e487fc128-metrics-client-ca\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.205474 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.205116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/388ad4ed-8180-4947-b4de-b09e487fc128-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.205474 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.205146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/388ad4ed-8180-4947-b4de-b09e487fc128-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.205474 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.205196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/388ad4ed-8180-4947-b4de-b09e487fc128-serving-certs-ca-bundle\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.205474 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.205236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flltq\" (UniqueName: \"kubernetes.io/projected/388ad4ed-8180-4947-b4de-b09e487fc128-kube-api-access-flltq\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.206661 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.206024    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/388ad4ed-8180-4947-b4de-b09e487fc128-serving-certs-ca-bundle\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.206661 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.206311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/388ad4ed-8180-4947-b4de-b09e487fc128-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.206661 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.206610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/388ad4ed-8180-4947-b4de-b09e487fc128-metrics-client-ca\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.208145 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.208122    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/388ad4ed-8180-4947-b4de-b09e487fc128-federate-client-tls\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.208257 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.208197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/388ad4ed-8180-4947-b4de-b09e487fc128-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.208365 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.208348    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/388ad4ed-8180-4947-b4de-b09e487fc128-telemeter-client-tls\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.208415 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.208394    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/388ad4ed-8180-4947-b4de-b09e487fc128-secret-telemeter-client\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.214038 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.214021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flltq\" (UniqueName: \"kubernetes.io/projected/388ad4ed-8180-4947-b4de-b09e487fc128-kube-api-access-flltq\") pod \"telemeter-client-cd85757d-vlm7b\" (UID: \"388ad4ed-8180-4947-b4de-b09e487fc128\") " pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.306330 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.306236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a2015b61-085f-4832-b797-139af3e5d12b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-npsd7\" (UID: \"a2015b61-085f-4832-b797-139af3e5d12b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-npsd7"
Jul 24 17:44:28.306484 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:44:28.306395    2578 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 24 17:44:28.306484 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:44:28.306470    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2015b61-085f-4832-b797-139af3e5d12b-monitoring-plugin-cert podName:a2015b61-085f-4832-b797-139af3e5d12b nodeName:}" failed. No retries permitted until 2026-07-24 17:44:29.30645337 +0000 UTC m=+91.829811826 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a2015b61-085f-4832-b797-139af3e5d12b-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-npsd7" (UID: "a2015b61-085f-4832-b797-139af3e5d12b") : secret "monitoring-plugin-cert" not found
Jul 24 17:44:28.362688 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.362657    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b"
Jul 24 17:44:28.396670 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.396640    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:28.952725 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.952689    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:28.955716 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.954085    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:44:28.962027 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.961997    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:28.964858 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.964834    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 24 17:44:28.965062 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.965031    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 24 17:44:28.965137 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.965068    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 24 17:44:28.965207 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.964837    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 24 17:44:28.965387 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.965369    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 24 17:44:28.965624 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.965604    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vtcq2\""
Jul 24 17:44:28.965624 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.965610    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 24 17:44:28.965762 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.965609    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 24 17:44:28.966186 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.966159    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 24 17:44:28.969005 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.968892    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 24 17:44:28.969005 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.968977    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 24 17:44:28.969005 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.968989    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-eeus4q2ljg2bg\""
Jul 24 17:44:28.969198 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.969082    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 24 17:44:28.969198 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.968977    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 24 17:44:28.969301 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.969273    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 24 17:44:28.970893 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.970875    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 24 17:44:28.975017 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:28.974994    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:44:29.013412 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.013203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97bcd8e2-25db-4ef1-a41b-9584825b896b-config-out\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.013412 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.013268    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.013412 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.013296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97bcd8e2-25db-4ef1-a41b-9584825b896b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.013412 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.013321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.013412 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.013363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.014648 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.013449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.014648 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.013491    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.014648 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.014474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.014648 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.014518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwkf9\" (UniqueName: \"kubernetes.io/projected/97bcd8e2-25db-4ef1-a41b-9584825b896b-kube-api-access-jwkf9\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.014648 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.014562    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.014648 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.014589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.014648 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.014621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.014648 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.014644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-web-config\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.015081 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.014668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.015081 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.014712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.015081 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.014823    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-config\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.015081 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.014856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.015081 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.014919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.015081 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.014959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.015081 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.014991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.115865 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.115822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.116047 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.115881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.116047 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.115916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97bcd8e2-25db-4ef1-a41b-9584825b896b-config-out\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.116047 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.115962    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.116047 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.115993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97bcd8e2-25db-4ef1-a41b-9584825b896b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.116047 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.116014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.116308 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.116062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.116308 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.116096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.116308 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.116140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.116308 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.116203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.116308 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.116234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwkf9\" (UniqueName: \"kubernetes.io/projected/97bcd8e2-25db-4ef1-a41b-9584825b896b-kube-api-access-jwkf9\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.116308 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.116279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.116591 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.116312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.116591 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.116348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.116591 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.116373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-web-config\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.116591 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.116402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.116591 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.116429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.116591 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.116458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-config\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.116591 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.116484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.116591 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.116531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.117568 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.117391    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.117698 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.117406    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.123415 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.118716    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.123415 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.119513    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.123415 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.120153    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.123415 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.122496    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.123415 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.122987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.123415 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.123373    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.124453 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.124407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.125015 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.124687    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.125015 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.124977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-web-config\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.125751 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.125726    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.125906 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.125825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97bcd8e2-25db-4ef1-a41b-9584825b896b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.126557 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.126118    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97bcd8e2-25db-4ef1-a41b-9584825b896b-config-out\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.126557 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.126179    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-config\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.126557 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.126275    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.126557 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.126515    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.126827 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.126688    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.128504 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.128462    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.129402 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.129366    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwkf9\" (UniqueName: \"kubernetes.io/projected/97bcd8e2-25db-4ef1-a41b-9584825b896b-kube-api-access-jwkf9\") pod \"prometheus-k8s-0\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.217186 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.217098    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:29.279948 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.279655    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:29.314818 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.314774    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:29.318036 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.318006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a2015b61-085f-4832-b797-139af3e5d12b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-npsd7\" (UID: \"a2015b61-085f-4832-b797-139af3e5d12b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-npsd7"
Jul 24 17:44:29.322589 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.322537    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a2015b61-085f-4832-b797-139af3e5d12b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-npsd7\" (UID: \"a2015b61-085f-4832-b797-139af3e5d12b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-npsd7"
Jul 24 17:44:29.400887 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:29.400845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-npsd7"
Jul 24 17:44:32.298562 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:32.298531    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:32.383066 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:32.383029    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:34.701094 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:34.697578    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-cd85757d-vlm7b"]
Jul 24 17:44:34.724936 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:34.724759    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bc9895498-jf4x5"]
Jul 24 17:44:34.905181 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:44:34.905079    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388ad4ed_8180_4947_b4de_b09e487fc128.slice/crio-cc71ba6a23f30e44582312909810a1347960d32c139e8f98bc62630996123cd0 WatchSource:0}: Error finding container cc71ba6a23f30e44582312909810a1347960d32c139e8f98bc62630996123cd0: Status 404 returned error can't find the container with id cc71ba6a23f30e44582312909810a1347960d32c139e8f98bc62630996123cd0
Jul 24 17:44:34.907136 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:44:34.907108    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419c9f71_0a9d_4a42_b11d_e3eebaa895a1.slice/crio-ba3f985b5bfb2ba305a2599511630a2ea8b2fac5f5ae396a567222f9ee941d0f WatchSource:0}: Error finding container ba3f985b5bfb2ba305a2599511630a2ea8b2fac5f5ae396a567222f9ee941d0f: Status 404 returned error can't find the container with id ba3f985b5bfb2ba305a2599511630a2ea8b2fac5f5ae396a567222f9ee941d0f
Jul 24 17:44:34.955626 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:34.955598    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"]
Jul 24 17:44:34.959844 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:34.959516    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-npsd7"]
Jul 24 17:44:34.966497 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:34.965957    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:44:34.974275 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:44:34.973692    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97bcd8e2_25db_4ef1_a41b_9584825b896b.slice/crio-03779d7b0a34fa7e4af4603152c4a4c16b3cd5a08dd38cdbbd5daaca2491f84d WatchSource:0}: Error finding container 03779d7b0a34fa7e4af4603152c4a4c16b3cd5a08dd38cdbbd5daaca2491f84d: Status 404 returned error can't find the container with id 03779d7b0a34fa7e4af4603152c4a4c16b3cd5a08dd38cdbbd5daaca2491f84d
Jul 24 17:44:35.354261 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:35.354224    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl" event={"ID":"2660087a-bcaa-41c3-b3e3-12070c59fe76","Type":"ContainerStarted","Data":"684c834f755f59e4083152b45bab0b25cfbbe1a731cb62b9841d99c4132bcd6a"}
Jul 24 17:44:35.355708 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:35.355671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5" event={"ID":"419c9f71-0a9d-4a42-b11d-e3eebaa895a1","Type":"ContainerStarted","Data":"ba3f985b5bfb2ba305a2599511630a2ea8b2fac5f5ae396a567222f9ee941d0f"}
Jul 24 17:44:35.357465 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:35.357431    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-kx42n" event={"ID":"0a303a45-e8f0-4b53-940b-d22569b73afb","Type":"ContainerStarted","Data":"63ba2349c93670d9fa499f0b4e712cad6c220cc9329c5437241ef39631c086e9"}
Jul 24 17:44:35.357654 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:35.357613    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-kx42n"
Jul 24 17:44:35.359376 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:35.359307    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pfqcd" event={"ID":"d8d0a6d2-c541-427e-b1e2-48fa4db65735","Type":"ContainerStarted","Data":"450d5ca0713b84e115c9a732256396a139ced2b04e007f8eb8c72b94082dfba1"}
Jul 24 17:44:35.361088 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:35.361065    2578 generic.go:356] "Generic (PLEG): container finished" podID="556b6001-3830-478a-afd7-9cd9bc3a9926" containerID="228648b9303461bfac510167d88228f70d4c2e6513c1ca113c5303d498bc5522" exitCode=0
Jul 24 17:44:35.361182 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:35.361130    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfvzx" event={"ID":"556b6001-3830-478a-afd7-9cd9bc3a9926","Type":"ContainerDied","Data":"228648b9303461bfac510167d88228f70d4c2e6513c1ca113c5303d498bc5522"}
Jul 24 17:44:35.362411 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:35.362388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-npsd7" event={"ID":"a2015b61-085f-4832-b797-139af3e5d12b","Type":"ContainerStarted","Data":"6f20c7ee2e118661dd617dd8a0652f1fc7768dc713bf48ae049c9d565770412e"}
Jul 24 17:44:35.363686 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:35.363655    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b" event={"ID":"388ad4ed-8180-4947-b4de-b09e487fc128","Type":"ContainerStarted","Data":"cc71ba6a23f30e44582312909810a1347960d32c139e8f98bc62630996123cd0"}
Jul 24 17:44:35.365006 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:35.364983    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97bcd8e2-25db-4ef1-a41b-9584825b896b","Type":"ContainerStarted","Data":"03779d7b0a34fa7e4af4603152c4a4c16b3cd5a08dd38cdbbd5daaca2491f84d"}
Jul 24 17:44:35.371253 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:35.371226    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-kx42n"
Jul 24 17:44:35.393831 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:35.393645    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-kx42n" podStartSLOduration=2.053071411 podStartE2EDuration="20.393626031s" totalImagesPullingTime="18.34055462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:44:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:35.374496717 +0000 UTC m=+97.897855196" watchObservedRunningTime="2026-07-24 17:44:35.393626031 +0000 UTC m=+97.916984512"
Jul 24 17:44:35.799048 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:35.799011    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:44:36.183639 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:36.183560    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:36.280939 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:36.280846    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:36.377057 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:36.376940    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfvzx" event={"ID":"556b6001-3830-478a-afd7-9cd9bc3a9926","Type":"ContainerStarted","Data":"ab8b9c1202fc4bf2a8e1127631d3f2b7bf27cb0cee20e26584cec8246ca453f2"}
Jul 24 17:44:36.377057 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:36.377000    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfvzx" event={"ID":"556b6001-3830-478a-afd7-9cd9bc3a9926","Type":"ContainerStarted","Data":"b0f8490a223c88b24b74c2628885587acbaf3da84a8a5d716e709f27d808bb24"}
Jul 24 17:44:36.380453 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:36.380374    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:36.401256 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:36.401023    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pfvzx" podStartSLOduration=3.147440298 podStartE2EDuration="14.400999348s" totalImagesPullingTime="11.175199191s" totalInitContainerRuntime="78.359859ms" isStatefulPod=true podCreationTimestamp="2026-07-24 17:44:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:36.397623431 +0000 UTC m=+98.920981909" watchObservedRunningTime="2026-07-24 17:44:36.400999348 +0000 UTC m=+98.924357845"
Jul 24 17:44:38.279606 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:38.279575    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-657dd9f654-2fvxt"
Jul 24 17:44:38.281407 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:38.281379    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:38.365146 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:38.365067    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:38.435118 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:38.435036    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:38.536176 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:38.536102    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:40.808806 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:40.808750    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv" podUID="33baa17d-da25-4f28-881f-55035131c237" containerName="registry" containerID="cri-o://ef5b980dff801ddeaa6590b3f0295d1177a49a8bd211d155a7dc248f13f9b135" gracePeriod=30
Jul 24 17:44:41.245162 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:41.245076    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8g49q"
Jul 24 17:44:41.374417 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:41.374385    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:41.431803 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:41.431751    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:43.303952 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:43.303920    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:43.404569 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:43.404533    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:43.409365 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:43.409326    2578 generic.go:356] "Generic (PLEG): container finished" podID="33baa17d-da25-4f28-881f-55035131c237" containerID="ef5b980dff801ddeaa6590b3f0295d1177a49a8bd211d155a7dc248f13f9b135" exitCode=0
Jul 24 17:44:43.409514 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:43.409390    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv" event={"ID":"33baa17d-da25-4f28-881f-55035131c237","Type":"ContainerDied","Data":"ef5b980dff801ddeaa6590b3f0295d1177a49a8bd211d155a7dc248f13f9b135"}
Jul 24 17:44:43.871426 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:43.871391    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:43.927217 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:43.927187    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:44.041122 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.041087    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:44:44.178460 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.178408    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33baa17d-da25-4f28-881f-55035131c237-registry-certificates\") pod \"33baa17d-da25-4f28-881f-55035131c237\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") "
Jul 24 17:44:44.179553 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.178562    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-bound-sa-token\") pod \"33baa17d-da25-4f28-881f-55035131c237\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") "
Jul 24 17:44:44.179553 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.178614    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls\") pod \"33baa17d-da25-4f28-881f-55035131c237\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") "
Jul 24 17:44:44.179553 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.178641    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzhkj\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-kube-api-access-zzhkj\") pod \"33baa17d-da25-4f28-881f-55035131c237\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") "
Jul 24 17:44:44.179553 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.178675    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33baa17d-da25-4f28-881f-55035131c237-installation-pull-secrets\") pod \"33baa17d-da25-4f28-881f-55035131c237\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") "
Jul 24 17:44:44.179553 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.178714    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33baa17d-da25-4f28-881f-55035131c237-trusted-ca\") pod \"33baa17d-da25-4f28-881f-55035131c237\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") "
Jul 24 17:44:44.179553 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.178744    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33baa17d-da25-4f28-881f-55035131c237-image-registry-private-configuration\") pod \"33baa17d-da25-4f28-881f-55035131c237\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") "
Jul 24 17:44:44.179553 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.178945    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33baa17d-da25-4f28-881f-55035131c237-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "33baa17d-da25-4f28-881f-55035131c237" (UID: "33baa17d-da25-4f28-881f-55035131c237"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:44:44.179553 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.178962    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33baa17d-da25-4f28-881f-55035131c237-ca-trust-extracted\") pod \"33baa17d-da25-4f28-881f-55035131c237\" (UID: \"33baa17d-da25-4f28-881f-55035131c237\") "
Jul 24 17:44:44.179553 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.179281    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33baa17d-da25-4f28-881f-55035131c237-registry-certificates\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:44:44.180325 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.179679    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33baa17d-da25-4f28-881f-55035131c237-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "33baa17d-da25-4f28-881f-55035131c237" (UID: "33baa17d-da25-4f28-881f-55035131c237"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:44:44.183178 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.183134    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "33baa17d-da25-4f28-881f-55035131c237" (UID: "33baa17d-da25-4f28-881f-55035131c237"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:44:44.183657 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.183477    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-kube-api-access-zzhkj" (OuterVolumeSpecName: "kube-api-access-zzhkj") pod "33baa17d-da25-4f28-881f-55035131c237" (UID: "33baa17d-da25-4f28-881f-55035131c237"). InnerVolumeSpecName "kube-api-access-zzhkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:44:44.183657 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.183602    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33baa17d-da25-4f28-881f-55035131c237-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "33baa17d-da25-4f28-881f-55035131c237" (UID: "33baa17d-da25-4f28-881f-55035131c237"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:44:44.183964 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.183876    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "33baa17d-da25-4f28-881f-55035131c237" (UID: "33baa17d-da25-4f28-881f-55035131c237"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:44:44.186637 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.186602    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33baa17d-da25-4f28-881f-55035131c237-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "33baa17d-da25-4f28-881f-55035131c237" (UID: "33baa17d-da25-4f28-881f-55035131c237"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:44:44.195826 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.193325    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33baa17d-da25-4f28-881f-55035131c237-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "33baa17d-da25-4f28-881f-55035131c237" (UID: "33baa17d-da25-4f28-881f-55035131c237"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:44:44.280074 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.279988    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33baa17d-da25-4f28-881f-55035131c237-ca-trust-extracted\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:44:44.280074 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.280014    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-bound-sa-token\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:44:44.280074 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.280024    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-registry-tls\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:44:44.280074 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.280031    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzhkj\" (UniqueName: \"kubernetes.io/projected/33baa17d-da25-4f28-881f-55035131c237-kube-api-access-zzhkj\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:44:44.280074 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.280040    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33baa17d-da25-4f28-881f-55035131c237-installation-pull-secrets\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:44:44.280074 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.280048    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33baa17d-da25-4f28-881f-55035131c237-trusted-ca\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:44:44.280074 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.280056    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33baa17d-da25-4f28-881f-55035131c237-image-registry-private-configuration\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:44:44.415924 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.415892    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl" event={"ID":"2660087a-bcaa-41c3-b3e3-12070c59fe76","Type":"ContainerStarted","Data":"56f8c125a60aebc9745bfefaeac8f08cc60ebb6c218381054a310a59180ced05"}
Jul 24 17:44:44.418057 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.418031    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5" event={"ID":"419c9f71-0a9d-4a42-b11d-e3eebaa895a1","Type":"ContainerStarted","Data":"56817661f5877fd10d58e12d5859dbc13ea024e815c20294df921034e62ea950"}
Jul 24 17:44:44.418155 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.418100    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5" event={"ID":"419c9f71-0a9d-4a42-b11d-e3eebaa895a1","Type":"ContainerStarted","Data":"e23a01c21c2c8ce58a0ef846ea796caaa65bac869e948f7991760c2da1c45049"}
Jul 24 17:44:44.419343 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.419323    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv"
Jul 24 17:44:44.419652 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.419631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7df8c856dc-jqsbv" event={"ID":"33baa17d-da25-4f28-881f-55035131c237","Type":"ContainerDied","Data":"b265c8bdbb2dd0f3b42fa367a7efc808ac16890efc9d2ce3d480da1fba1f5b73"}
Jul 24 17:44:44.419709 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.419676    2578 scope.go:117] "RemoveContainer" containerID="ef5b980dff801ddeaa6590b3f0295d1177a49a8bd211d155a7dc248f13f9b135"
Jul 24 17:44:44.421831 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.421774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-npsd7" event={"ID":"a2015b61-085f-4832-b797-139af3e5d12b","Type":"ContainerStarted","Data":"7c96161deec8a8f6dfb3934770b31cc4e93572a335d903ba8d26be15008aa22a"}
Jul 24 17:44:44.422006 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.421965    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-npsd7"
Jul 24 17:44:44.424033 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.424012    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b" event={"ID":"388ad4ed-8180-4947-b4de-b09e487fc128","Type":"ContainerStarted","Data":"f17c41e2627f788e2d670311b561cbdba929169b43b7502b002696e4bbc8ec63"}
Jul 24 17:44:44.426182 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.426160    2578 generic.go:356] "Generic (PLEG): container finished" podID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerID="c1bf92f228faa7d0bfcc5f3b491a9f63dc3fd18c86400adfcd1a2f66482c8a21" exitCode=0
Jul 24 17:44:44.426287 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.426195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97bcd8e2-25db-4ef1-a41b-9584825b896b","Type":"ContainerDied","Data":"c1bf92f228faa7d0bfcc5f3b491a9f63dc3fd18c86400adfcd1a2f66482c8a21"}
Jul 24 17:44:44.428670 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.428619    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-npsd7"
Jul 24 17:44:44.433544 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.433506    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl" podStartSLOduration=8.39055768 podStartE2EDuration="17.433493254s" totalImagesPullingTime="9.042935574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:44:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:44.43234559 +0000 UTC m=+106.955704067" watchObservedRunningTime="2026-07-24 17:44:44.433493254 +0000 UTC m=+106.956851732"
Jul 24 17:44:44.452333 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.452288    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-npsd7" podStartSLOduration=8.408833242 podStartE2EDuration="17.452270627s" totalImagesPullingTime="9.043437385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:44:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:44.452165352 +0000 UTC m=+106.975523857" watchObservedRunningTime="2026-07-24 17:44:44.452270627 +0000 UTC m=+106.975629105"
Jul 24 17:44:44.500626 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.500596    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7df8c856dc-jqsbv"]
Jul 24 17:44:44.504223 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:44.504196    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7df8c856dc-jqsbv"]
Jul 24 17:44:45.434939 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:45.434822    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pfqcd" event={"ID":"d8d0a6d2-c541-427e-b1e2-48fa4db65735","Type":"ContainerStarted","Data":"2dd6bce65d7895613c54b352e56f078e7ddb931be377bc476c06a2b76a132fe7"}
Jul 24 17:44:45.438155 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:45.438122    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b" event={"ID":"388ad4ed-8180-4947-b4de-b09e487fc128","Type":"ContainerStarted","Data":"ee49cb8913124881e1427615d21663c13c41fc408ba18b7e274457206a1ee7b1"}
Jul 24 17:44:45.438297 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:45.438188    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b" event={"ID":"388ad4ed-8180-4947-b4de-b09e487fc128","Type":"ContainerStarted","Data":"b3e6371fccdc2107bca79c1deaf4e9f91a2ff3997dc603d2868bac9a1dc6947a"}
Jul 24 17:44:45.441352 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:45.441301    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5" event={"ID":"419c9f71-0a9d-4a42-b11d-e3eebaa895a1","Type":"ContainerStarted","Data":"dba6ba2e776ccbf6e5afba89797adddfd40fc6d776bc457fc7ffe2ff61061658"}
Jul 24 17:44:45.461184 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:45.461134    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-cd85757d-vlm7b" podStartSLOduration=8.361044798 podStartE2EDuration="17.461116919s" totalImagesPullingTime="9.100072121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:45.46011212 +0000 UTC m=+107.983470624" watchObservedRunningTime="2026-07-24 17:44:45.461116919 +0000 UTC m=+107.984475395"
Jul 24 17:44:45.969483 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:45.969450    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33baa17d-da25-4f28-881f-55035131c237" path="/var/lib/kubelet/pods/33baa17d-da25-4f28-881f-55035131c237/volumes"
Jul 24 17:44:46.452386 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:46.452283    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5" event={"ID":"419c9f71-0a9d-4a42-b11d-e3eebaa895a1","Type":"ContainerStarted","Data":"225c9308d577a08c1d1f3b389fdf3c4c00094d18a1ad2ea6998fdf7a2281a66a"}
Jul 24 17:44:46.452386 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:46.452367    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5" event={"ID":"419c9f71-0a9d-4a42-b11d-e3eebaa895a1","Type":"ContainerStarted","Data":"5cf853d8e5ee278e224014c276f0b4ec996529cd04c83ca1ace17e2650b93032"}
Jul 24 17:44:46.452933 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:46.452408    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5" event={"ID":"419c9f71-0a9d-4a42-b11d-e3eebaa895a1","Type":"ContainerStarted","Data":"7d0d86f13cfcb7ed54e7a196075e65d14cce90ab3c8c61fb2f2b06382463f0aa"}
Jul 24 17:44:46.476794 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:46.476694    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5" podStartSLOduration=11.019241954 podStartE2EDuration="21.476674073s" totalImagesPullingTime="10.457432119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:44:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:46.475108702 +0000 UTC m=+108.998467207" watchObservedRunningTime="2026-07-24 17:44:46.476674073 +0000 UTC m=+109.000032552"
Jul 24 17:44:47.419915 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:47.419881    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:47.420102 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:47.419947    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:44:47.457127 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:47.457099    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:50.471898 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:50.471869    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pfqcd" event={"ID":"d8d0a6d2-c541-427e-b1e2-48fa4db65735","Type":"ContainerStarted","Data":"9cf620124e919ea0775bede696a96aa77086a3952bbc89c8febb8c5206f6ddcf"}
Jul 24 17:44:50.474828 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:50.474803    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97bcd8e2-25db-4ef1-a41b-9584825b896b","Type":"ContainerStarted","Data":"8e2e365bfcff5f6aa944b29d7851cf44e11ac652b2aaaf53a3181bc9ab0a695f"}
Jul 24 17:44:50.474904 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:50.474862    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97bcd8e2-25db-4ef1-a41b-9584825b896b","Type":"ContainerStarted","Data":"aaa0c2891cfdc9163b9dc38dc236262f495d84f57b65b65679f755abbc42fa74"}
Jul 24 17:44:50.474904 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:50.474889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97bcd8e2-25db-4ef1-a41b-9584825b896b","Type":"ContainerStarted","Data":"fd3fb6ad2d9e3f0af66e54e6a3f8298b888c86a8cda15d71aca0e66f063acc2f"}
Jul 24 17:44:50.474977 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:50.474914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97bcd8e2-25db-4ef1-a41b-9584825b896b","Type":"ContainerStarted","Data":"74f991f378d0d079cb95cf763ea61053dcf13b6c1f2e3038b80b6b7b08805245"}
Jul 24 17:44:50.501138 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:50.501092    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pfqcd" podStartSLOduration=34.060644126 podStartE2EDuration="1m2.501076283s" totalImagesPullingTime="28.440432157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:43:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:50.500574415 +0000 UTC m=+113.023932894" watchObservedRunningTime="2026-07-24 17:44:50.501076283 +0000 UTC m=+113.024434765"
Jul 24 17:44:51.480931 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:51.480892    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97bcd8e2-25db-4ef1-a41b-9584825b896b","Type":"ContainerStarted","Data":"a1c115a2933647893f05b6b7dddeab295678e88efcb862fc1e9880ac9dd30a5c"}
Jul 24 17:44:51.481522 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:51.481295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97bcd8e2-25db-4ef1-a41b-9584825b896b","Type":"ContainerStarted","Data":"e414c50e011df4cbaa5ceefefff52d7ecf571c34595b1e1c26ee24a9343102e6"}
Jul 24 17:44:51.512116 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:51.512030    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=8.92016392 podStartE2EDuration="23.512009257s" totalImagesPullingTime="14.565282827s" totalInitContainerRuntime="26.56251ms" isStatefulPod=false podCreationTimestamp="2026-07-24 17:44:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:51.510245078 +0000 UTC m=+114.033603555" watchObservedRunningTime="2026-07-24 17:44:51.512009257 +0000 UTC m=+114.035367735"
Jul 24 17:44:53.467933 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:53.467906    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7bc9895498-jf4x5"
Jul 24 17:44:54.282311 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:54.282276    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:44:58.096661 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:58.096569    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:58.197260 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:58.197226    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:58.255135 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:58.255101    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:58.868372 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:58.868331    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:58.967944 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:44:58.967913    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:00.855356 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:00.855321    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:00.927870 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:00.927838    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:06.354579 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:06.354541    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:06.455592 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:06.455548    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:06.555105 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:06.555071    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:07.425050 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:07.425019    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:45:07.428681 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:07.428659    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-585dbd8bc9-5lsgl"
Jul 24 17:45:07.461383 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:07.461350    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:07.544773 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:07.544744    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:07.706521 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:07.706442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs\") pod \"network-metrics-daemon-d4gbz\" (UID: \"64e0f517-9969-47f1-8daf-8dfa194ba923\") " pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:45:07.708847 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:07.708825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e0f517-9969-47f1-8daf-8dfa194ba923-metrics-certs\") pod \"network-metrics-daemon-d4gbz\" (UID: \"64e0f517-9969-47f1-8daf-8dfa194ba923\") " pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:45:07.878109 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:07.878079    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pdhgn\""
Jul 24 17:45:07.886439 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:07.886417    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d4gbz"
Jul 24 17:45:08.046428 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:08.046405    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d4gbz"]
Jul 24 17:45:08.048763 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:45:08.048739    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e0f517_9969_47f1_8daf_8dfa194ba923.slice/crio-acc6d1c9a42326078895babf7f3c00e97514d3a9bc8e775197b3eebb90c9866a WatchSource:0}: Error finding container acc6d1c9a42326078895babf7f3c00e97514d3a9bc8e775197b3eebb90c9866a: Status 404 returned error can't find the container with id acc6d1c9a42326078895babf7f3c00e97514d3a9bc8e775197b3eebb90c9866a
Jul 24 17:45:08.543138 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:08.543104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d4gbz" event={"ID":"64e0f517-9969-47f1-8daf-8dfa194ba923","Type":"ContainerStarted","Data":"acc6d1c9a42326078895babf7f3c00e97514d3a9bc8e775197b3eebb90c9866a"}
Jul 24 17:45:10.552905 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:10.552871    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d4gbz" event={"ID":"64e0f517-9969-47f1-8daf-8dfa194ba923","Type":"ContainerStarted","Data":"414736b2699a59af0a104016dc40469861fca82e03020f9f2e6f504c275d7336"}
Jul 24 17:45:10.553292 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:10.552928    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d4gbz" event={"ID":"64e0f517-9969-47f1-8daf-8dfa194ba923","Type":"ContainerStarted","Data":"ebe6722d5b09a46c88aa693b2bd61ca13230bb089648fa9be6d30e12c60e886c"}
Jul 24 17:45:10.572587 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:10.572535    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-d4gbz" podStartSLOduration=131.083008247 podStartE2EDuration="2m12.572521299s" totalImagesPullingTime="1.489513052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:45:10.570683962 +0000 UTC m=+133.094042453" watchObservedRunningTime="2026-07-24 17:45:10.572521299 +0000 UTC m=+133.095879776"
Jul 24 17:45:10.707216 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:10.707181    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:10.831998 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:10.831923    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:13.314496 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:13.314467    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:29.280320 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:29.280280    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:29.354167 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:29.354138    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:29.638938 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:29.638912    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:38.851017 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:38.850979    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:39.222039 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:39.221958    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:41.194790 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:41.194761    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:41.264310 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:41.264279    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:47.445826 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:47.445771    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:45:47.446305 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:47.446187    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="kube-rbac-proxy-web" containerID="cri-o://8e2e365bfcff5f6aa944b29d7851cf44e11ac652b2aaaf53a3181bc9ab0a695f" gracePeriod=600
Jul 24 17:45:47.446305 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:47.446187    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="thanos-sidecar" containerID="cri-o://aaa0c2891cfdc9163b9dc38dc236262f495d84f57b65b65679f755abbc42fa74" gracePeriod=600
Jul 24 17:45:47.446305 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:47.446208    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a1c115a2933647893f05b6b7dddeab295678e88efcb862fc1e9880ac9dd30a5c" gracePeriod=600
Jul 24 17:45:47.446305 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:47.446215    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="config-reloader" containerID="cri-o://fd3fb6ad2d9e3f0af66e54e6a3f8298b888c86a8cda15d71aca0e66f063acc2f" gracePeriod=600
Jul 24 17:45:47.446528 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:47.446395    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="kube-rbac-proxy" containerID="cri-o://e414c50e011df4cbaa5ceefefff52d7ecf571c34595b1e1c26ee24a9343102e6" gracePeriod=600
Jul 24 17:45:47.446528 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:47.446152    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="prometheus" containerID="cri-o://74f991f378d0d079cb95cf763ea61053dcf13b6c1f2e3038b80b6b7b08805245" gracePeriod=600
Jul 24 17:45:47.698359 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:47.698266    2578 generic.go:356] "Generic (PLEG): container finished" podID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerID="a1c115a2933647893f05b6b7dddeab295678e88efcb862fc1e9880ac9dd30a5c" exitCode=0
Jul 24 17:45:47.698359 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:47.698290    2578 generic.go:356] "Generic (PLEG): container finished" podID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerID="e414c50e011df4cbaa5ceefefff52d7ecf571c34595b1e1c26ee24a9343102e6" exitCode=0
Jul 24 17:45:47.698359 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:47.698297    2578 generic.go:356] "Generic (PLEG): container finished" podID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerID="aaa0c2891cfdc9163b9dc38dc236262f495d84f57b65b65679f755abbc42fa74" exitCode=0
Jul 24 17:45:47.698359 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:47.698304    2578 generic.go:356] "Generic (PLEG): container finished" podID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerID="fd3fb6ad2d9e3f0af66e54e6a3f8298b888c86a8cda15d71aca0e66f063acc2f" exitCode=0
Jul 24 17:45:47.698359 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:47.698312    2578 generic.go:356] "Generic (PLEG): container finished" podID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerID="74f991f378d0d079cb95cf763ea61053dcf13b6c1f2e3038b80b6b7b08805245" exitCode=0
Jul 24 17:45:47.698359 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:47.698344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97bcd8e2-25db-4ef1-a41b-9584825b896b","Type":"ContainerDied","Data":"a1c115a2933647893f05b6b7dddeab295678e88efcb862fc1e9880ac9dd30a5c"}
Jul 24 17:45:47.698797 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:47.698409    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97bcd8e2-25db-4ef1-a41b-9584825b896b","Type":"ContainerDied","Data":"e414c50e011df4cbaa5ceefefff52d7ecf571c34595b1e1c26ee24a9343102e6"}
Jul 24 17:45:47.698797 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:47.698435    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97bcd8e2-25db-4ef1-a41b-9584825b896b","Type":"ContainerDied","Data":"aaa0c2891cfdc9163b9dc38dc236262f495d84f57b65b65679f755abbc42fa74"}
Jul 24 17:45:47.698797 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:47.698459    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97bcd8e2-25db-4ef1-a41b-9584825b896b","Type":"ContainerDied","Data":"fd3fb6ad2d9e3f0af66e54e6a3f8298b888c86a8cda15d71aca0e66f063acc2f"}
Jul 24 17:45:47.698797 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:47.698483    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97bcd8e2-25db-4ef1-a41b-9584825b896b","Type":"ContainerDied","Data":"74f991f378d0d079cb95cf763ea61053dcf13b6c1f2e3038b80b6b7b08805245"}
Jul 24 17:45:48.691652 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.691630    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:48.704675 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.704653    2578 generic.go:356] "Generic (PLEG): container finished" podID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerID="8e2e365bfcff5f6aa944b29d7851cf44e11ac652b2aaaf53a3181bc9ab0a695f" exitCode=0
Jul 24 17:45:48.704792 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.704701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97bcd8e2-25db-4ef1-a41b-9584825b896b","Type":"ContainerDied","Data":"8e2e365bfcff5f6aa944b29d7851cf44e11ac652b2aaaf53a3181bc9ab0a695f"}
Jul 24 17:45:48.704792 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.704733    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:48.704869 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.704815    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97bcd8e2-25db-4ef1-a41b-9584825b896b","Type":"ContainerDied","Data":"03779d7b0a34fa7e4af4603152c4a4c16b3cd5a08dd38cdbbd5daaca2491f84d"}
Jul 24 17:45:48.704869 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.704850    2578 scope.go:117] "RemoveContainer" containerID="a1c115a2933647893f05b6b7dddeab295678e88efcb862fc1e9880ac9dd30a5c"
Jul 24 17:45:48.711875 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.711855    2578 scope.go:117] "RemoveContainer" containerID="e414c50e011df4cbaa5ceefefff52d7ecf571c34595b1e1c26ee24a9343102e6"
Jul 24 17:45:48.718562 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.718538    2578 scope.go:117] "RemoveContainer" containerID="8e2e365bfcff5f6aa944b29d7851cf44e11ac652b2aaaf53a3181bc9ab0a695f"
Jul 24 17:45:48.730239 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.730217    2578 scope.go:117] "RemoveContainer" containerID="aaa0c2891cfdc9163b9dc38dc236262f495d84f57b65b65679f755abbc42fa74"
Jul 24 17:45:48.738975 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.738948    2578 scope.go:117] "RemoveContainer" containerID="fd3fb6ad2d9e3f0af66e54e6a3f8298b888c86a8cda15d71aca0e66f063acc2f"
Jul 24 17:45:48.747270 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.747250    2578 scope.go:117] "RemoveContainer" containerID="74f991f378d0d079cb95cf763ea61053dcf13b6c1f2e3038b80b6b7b08805245"
Jul 24 17:45:48.754610 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.754590    2578 scope.go:117] "RemoveContainer" containerID="c1bf92f228faa7d0bfcc5f3b491a9f63dc3fd18c86400adfcd1a2f66482c8a21"
Jul 24 17:45:48.761287 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.761270    2578 scope.go:117] "RemoveContainer" containerID="a1c115a2933647893f05b6b7dddeab295678e88efcb862fc1e9880ac9dd30a5c"
Jul 24 17:45:48.761686 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:45:48.761555    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c115a2933647893f05b6b7dddeab295678e88efcb862fc1e9880ac9dd30a5c\": container with ID starting with a1c115a2933647893f05b6b7dddeab295678e88efcb862fc1e9880ac9dd30a5c not found: ID does not exist" containerID="a1c115a2933647893f05b6b7dddeab295678e88efcb862fc1e9880ac9dd30a5c"
Jul 24 17:45:48.761686 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.761595    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c115a2933647893f05b6b7dddeab295678e88efcb862fc1e9880ac9dd30a5c"} err="failed to get container status \"a1c115a2933647893f05b6b7dddeab295678e88efcb862fc1e9880ac9dd30a5c\": rpc error: code = NotFound desc = could not find container \"a1c115a2933647893f05b6b7dddeab295678e88efcb862fc1e9880ac9dd30a5c\": container with ID starting with a1c115a2933647893f05b6b7dddeab295678e88efcb862fc1e9880ac9dd30a5c not found: ID does not exist"
Jul 24 17:45:48.761686 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.761640    2578 scope.go:117] "RemoveContainer" containerID="e414c50e011df4cbaa5ceefefff52d7ecf571c34595b1e1c26ee24a9343102e6"
Jul 24 17:45:48.762058 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:45:48.762038    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e414c50e011df4cbaa5ceefefff52d7ecf571c34595b1e1c26ee24a9343102e6\": container with ID starting with e414c50e011df4cbaa5ceefefff52d7ecf571c34595b1e1c26ee24a9343102e6 not found: ID does not exist" containerID="e414c50e011df4cbaa5ceefefff52d7ecf571c34595b1e1c26ee24a9343102e6"
Jul 24 17:45:48.762115 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.762066    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e414c50e011df4cbaa5ceefefff52d7ecf571c34595b1e1c26ee24a9343102e6"} err="failed to get container status \"e414c50e011df4cbaa5ceefefff52d7ecf571c34595b1e1c26ee24a9343102e6\": rpc error: code = NotFound desc = could not find container \"e414c50e011df4cbaa5ceefefff52d7ecf571c34595b1e1c26ee24a9343102e6\": container with ID starting with e414c50e011df4cbaa5ceefefff52d7ecf571c34595b1e1c26ee24a9343102e6 not found: ID does not exist"
Jul 24 17:45:48.762115 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.762087    2578 scope.go:117] "RemoveContainer" containerID="8e2e365bfcff5f6aa944b29d7851cf44e11ac652b2aaaf53a3181bc9ab0a695f"
Jul 24 17:45:48.762330 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:45:48.762313    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2e365bfcff5f6aa944b29d7851cf44e11ac652b2aaaf53a3181bc9ab0a695f\": container with ID starting with 8e2e365bfcff5f6aa944b29d7851cf44e11ac652b2aaaf53a3181bc9ab0a695f not found: ID does not exist" containerID="8e2e365bfcff5f6aa944b29d7851cf44e11ac652b2aaaf53a3181bc9ab0a695f"
Jul 24 17:45:48.762372 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.762335    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2e365bfcff5f6aa944b29d7851cf44e11ac652b2aaaf53a3181bc9ab0a695f"} err="failed to get container status \"8e2e365bfcff5f6aa944b29d7851cf44e11ac652b2aaaf53a3181bc9ab0a695f\": rpc error: code = NotFound desc = could not find container \"8e2e365bfcff5f6aa944b29d7851cf44e11ac652b2aaaf53a3181bc9ab0a695f\": container with ID starting with 8e2e365bfcff5f6aa944b29d7851cf44e11ac652b2aaaf53a3181bc9ab0a695f not found: ID does not exist"
Jul 24 17:45:48.762372 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.762352    2578 scope.go:117] "RemoveContainer" containerID="aaa0c2891cfdc9163b9dc38dc236262f495d84f57b65b65679f755abbc42fa74"
Jul 24 17:45:48.762611 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:45:48.762592    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa0c2891cfdc9163b9dc38dc236262f495d84f57b65b65679f755abbc42fa74\": container with ID starting with aaa0c2891cfdc9163b9dc38dc236262f495d84f57b65b65679f755abbc42fa74 not found: ID does not exist" containerID="aaa0c2891cfdc9163b9dc38dc236262f495d84f57b65b65679f755abbc42fa74"
Jul 24 17:45:48.762661 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.762617    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa0c2891cfdc9163b9dc38dc236262f495d84f57b65b65679f755abbc42fa74"} err="failed to get container status \"aaa0c2891cfdc9163b9dc38dc236262f495d84f57b65b65679f755abbc42fa74\": rpc error: code = NotFound desc = could not find container \"aaa0c2891cfdc9163b9dc38dc236262f495d84f57b65b65679f755abbc42fa74\": container with ID starting with aaa0c2891cfdc9163b9dc38dc236262f495d84f57b65b65679f755abbc42fa74 not found: ID does not exist"
Jul 24 17:45:48.762661 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.762646    2578 scope.go:117] "RemoveContainer" containerID="fd3fb6ad2d9e3f0af66e54e6a3f8298b888c86a8cda15d71aca0e66f063acc2f"
Jul 24 17:45:48.762924 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:45:48.762882    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd3fb6ad2d9e3f0af66e54e6a3f8298b888c86a8cda15d71aca0e66f063acc2f\": container with ID starting with fd3fb6ad2d9e3f0af66e54e6a3f8298b888c86a8cda15d71aca0e66f063acc2f not found: ID does not exist" containerID="fd3fb6ad2d9e3f0af66e54e6a3f8298b888c86a8cda15d71aca0e66f063acc2f"
Jul 24 17:45:48.763021 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.762931    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd3fb6ad2d9e3f0af66e54e6a3f8298b888c86a8cda15d71aca0e66f063acc2f"} err="failed to get container status \"fd3fb6ad2d9e3f0af66e54e6a3f8298b888c86a8cda15d71aca0e66f063acc2f\": rpc error: code = NotFound desc = could not find container \"fd3fb6ad2d9e3f0af66e54e6a3f8298b888c86a8cda15d71aca0e66f063acc2f\": container with ID starting with fd3fb6ad2d9e3f0af66e54e6a3f8298b888c86a8cda15d71aca0e66f063acc2f not found: ID does not exist"
Jul 24 17:45:48.763021 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.762947    2578 scope.go:117] "RemoveContainer" containerID="74f991f378d0d079cb95cf763ea61053dcf13b6c1f2e3038b80b6b7b08805245"
Jul 24 17:45:48.763201 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:45:48.763184    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74f991f378d0d079cb95cf763ea61053dcf13b6c1f2e3038b80b6b7b08805245\": container with ID starting with 74f991f378d0d079cb95cf763ea61053dcf13b6c1f2e3038b80b6b7b08805245 not found: ID does not exist" containerID="74f991f378d0d079cb95cf763ea61053dcf13b6c1f2e3038b80b6b7b08805245"
Jul 24 17:45:48.763242 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.763205    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f991f378d0d079cb95cf763ea61053dcf13b6c1f2e3038b80b6b7b08805245"} err="failed to get container status \"74f991f378d0d079cb95cf763ea61053dcf13b6c1f2e3038b80b6b7b08805245\": rpc error: code = NotFound desc = could not find container \"74f991f378d0d079cb95cf763ea61053dcf13b6c1f2e3038b80b6b7b08805245\": container with ID starting with 74f991f378d0d079cb95cf763ea61053dcf13b6c1f2e3038b80b6b7b08805245 not found: ID does not exist"
Jul 24 17:45:48.763242 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.763218    2578 scope.go:117] "RemoveContainer" containerID="c1bf92f228faa7d0bfcc5f3b491a9f63dc3fd18c86400adfcd1a2f66482c8a21"
Jul 24 17:45:48.763373 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:45:48.763359    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1bf92f228faa7d0bfcc5f3b491a9f63dc3fd18c86400adfcd1a2f66482c8a21\": container with ID starting with c1bf92f228faa7d0bfcc5f3b491a9f63dc3fd18c86400adfcd1a2f66482c8a21 not found: ID does not exist" containerID="c1bf92f228faa7d0bfcc5f3b491a9f63dc3fd18c86400adfcd1a2f66482c8a21"
Jul 24 17:45:48.763415 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.763375    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1bf92f228faa7d0bfcc5f3b491a9f63dc3fd18c86400adfcd1a2f66482c8a21"} err="failed to get container status \"c1bf92f228faa7d0bfcc5f3b491a9f63dc3fd18c86400adfcd1a2f66482c8a21\": rpc error: code = NotFound desc = could not find container \"c1bf92f228faa7d0bfcc5f3b491a9f63dc3fd18c86400adfcd1a2f66482c8a21\": container with ID starting with c1bf92f228faa7d0bfcc5f3b491a9f63dc3fd18c86400adfcd1a2f66482c8a21 not found: ID does not exist"
Jul 24 17:45:48.771711 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.771692    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-prometheus-k8s-tls\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.771798 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.771738    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97bcd8e2-25db-4ef1-a41b-9584825b896b-tls-assets\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.771798 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.771764    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-db\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.771885 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.771813    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-kube-rbac-proxy\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.771885 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.771841    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwkf9\" (UniqueName: \"kubernetes.io/projected/97bcd8e2-25db-4ef1-a41b-9584825b896b-kube-api-access-jwkf9\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.771885 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.771869    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-rulefiles-1\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.772024 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.771895    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-trusted-ca-bundle\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.772024 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.771935    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-configmap-kubelet-serving-ca-bundle\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.772024 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.771967    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.772024 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.772002    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97bcd8e2-25db-4ef1-a41b-9584825b896b-config-out\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.772220 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.772028    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-thanos-prometheus-http-client-file\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.772220 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.772073    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-configmap-metrics-client-ca\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.772220 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.772100    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.772220 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.772128    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-configmap-serving-certs-ca-bundle\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.772220 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.772168    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-rulefiles-2\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.772220 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.772197    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-web-config\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.772504 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.772222    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-rulefiles-0\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.772504 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.772261    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-config\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.772504 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.772290    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-grpc-tls\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.772504 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.772316    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-metrics-client-certs\") pod \"97bcd8e2-25db-4ef1-a41b-9584825b896b\" (UID: \"97bcd8e2-25db-4ef1-a41b-9584825b896b\") "
Jul 24 17:45:48.773251 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.773078    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:45:48.773352 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.773308    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:45:48.774821 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.773564    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:45:48.774821 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.773618    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:45:48.774821 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.773921    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:45:48.774821 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.774108    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:45:48.774821 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.774614    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:45:48.775167 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.775027    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:45:48.775167 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.775113    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:45:48.775476 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.775433    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97bcd8e2-25db-4ef1-a41b-9584825b896b-kube-api-access-jwkf9" (OuterVolumeSpecName: "kube-api-access-jwkf9") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "kube-api-access-jwkf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:45:48.776698 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.775970    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:45:48.776698 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.776040    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97bcd8e2-25db-4ef1-a41b-9584825b896b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:45:48.776698 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.776648    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:45:48.776893 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.776710    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bcd8e2-25db-4ef1-a41b-9584825b896b-config-out" (OuterVolumeSpecName: "config-out") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:45:48.777214 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.777191    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:45:48.777512 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.777487    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:45:48.777729 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.777707    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:45:48.777815 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.777723    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-config" (OuterVolumeSpecName: "config") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:45:48.777935 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.777913    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:45:48.787193 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.787170    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-web-config" (OuterVolumeSpecName: "web-config") pod "97bcd8e2-25db-4ef1-a41b-9584825b896b" (UID: "97bcd8e2-25db-4ef1-a41b-9584825b896b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:45:48.873348 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873317    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-configmap-metrics-client-ca\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:48.873348 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873343    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:48.873348 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873352    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:48.873538 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873364    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:48.873538 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873373    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-web-config\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:48.873538 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873381    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:48.873538 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873388    2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-config\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:48.873538 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873399    2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-grpc-tls\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:48.873538 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873407    2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-metrics-client-certs\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:48.873538 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873414    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:48.873538 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873422    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97bcd8e2-25db-4ef1-a41b-9584825b896b-tls-assets\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:48.873538 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873430    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-db\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:48.873538 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873437    2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-kube-rbac-proxy\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:48.873538 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873446    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwkf9\" (UniqueName: \"kubernetes.io/projected/97bcd8e2-25db-4ef1-a41b-9584825b896b-kube-api-access-jwkf9\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:48.873538 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873454    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:48.873538 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873464    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:48.873538 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873472    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bcd8e2-25db-4ef1-a41b-9584825b896b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:48.873538 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873481    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:48.873538 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873489    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97bcd8e2-25db-4ef1-a41b-9584825b896b-config-out\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:48.873538 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:48.873497    2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/97bcd8e2-25db-4ef1-a41b-9584825b896b-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:45:49.031559 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.031494    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:45:49.036127 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.036104    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:45:49.071438 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071413    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:45:49.071661 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071648    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="config-reloader"
Jul 24 17:45:49.071703 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071662    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="config-reloader"
Jul 24 17:45:49.071703 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071671    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="thanos-sidecar"
Jul 24 17:45:49.071703 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071678    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="thanos-sidecar"
Jul 24 17:45:49.071703 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071688    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="kube-rbac-proxy"
Jul 24 17:45:49.071703 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071694    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="kube-rbac-proxy"
Jul 24 17:45:49.071871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071705    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="init-config-reloader"
Jul 24 17:45:49.071871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071711    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="init-config-reloader"
Jul 24 17:45:49.071871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071718    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33baa17d-da25-4f28-881f-55035131c237" containerName="registry"
Jul 24 17:45:49.071871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071723    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="33baa17d-da25-4f28-881f-55035131c237" containerName="registry"
Jul 24 17:45:49.071871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071729    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="kube-rbac-proxy-web"
Jul 24 17:45:49.071871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071734    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="kube-rbac-proxy-web"
Jul 24 17:45:49.071871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071740    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="kube-rbac-proxy-thanos"
Jul 24 17:45:49.071871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071745    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="kube-rbac-proxy-thanos"
Jul 24 17:45:49.071871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071754    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="prometheus"
Jul 24 17:45:49.071871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071759    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="prometheus"
Jul 24 17:45:49.071871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071812    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="config-reloader"
Jul 24 17:45:49.071871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071820    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="kube-rbac-proxy"
Jul 24 17:45:49.071871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071827    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="prometheus"
Jul 24 17:45:49.071871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071833    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="thanos-sidecar"
Jul 24 17:45:49.071871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071838    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="33baa17d-da25-4f28-881f-55035131c237" containerName="registry"
Jul 24 17:45:49.071871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071845    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="kube-rbac-proxy-web"
Jul 24 17:45:49.071871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.071850    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" containerName="kube-rbac-proxy-thanos"
Jul 24 17:45:49.075098 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.075082    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.077761 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.077738    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 24 17:45:49.077956 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.077897    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 24 17:45:49.077956 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.077897    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 24 17:45:49.078073 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.078014    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 24 17:45:49.078247 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.078232    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 24 17:45:49.078368 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.078355    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 24 17:45:49.078597 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.078579    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 24 17:45:49.078652 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.078620    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 24 17:45:49.079403 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.079377    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 24 17:45:49.079403 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.079395    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 24 17:45:49.079741 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.079724    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 24 17:45:49.079827 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.079753    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 24 17:45:49.080290 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.080253    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vtcq2\""
Jul 24 17:45:49.080630 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.080615    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-eeus4q2ljg2bg\""
Jul 24 17:45:49.082474 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.082433    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 24 17:45:49.085844 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.085826    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 24 17:45:49.095592 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.095574    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:45:49.175923 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.175892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.175923 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.175926    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8b6c83c3-f823-49f5-9862-6103e98a970a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.176129 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.175950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.176129 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.175986    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.176129 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.176014    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.176129 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.176032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-config\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.176129 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.176053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.176129 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.176090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.176129 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.176111    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.176129 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.176129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc78g\" (UniqueName: \"kubernetes.io/projected/8b6c83c3-f823-49f5-9862-6103e98a970a-kube-api-access-hc78g\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.176390 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.176173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b6c83c3-f823-49f5-9862-6103e98a970a-config-out\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.176390 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.176198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.176390 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.176224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.176390 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.176245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b6c83c3-f823-49f5-9862-6103e98a970a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.176390 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.176260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.176390 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.176302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.176390 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.176327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.176390 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.176361    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-web-config\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.176390 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.176383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.176660 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.176404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.277658 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.277622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.277658 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.277660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.277921 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.277678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.277921 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.277697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-config\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.277921 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.277720    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.277921 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.277772    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.277921 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.277820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.277921 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.277849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc78g\" (UniqueName: \"kubernetes.io/projected/8b6c83c3-f823-49f5-9862-6103e98a970a-kube-api-access-hc78g\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.277921 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.277872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b6c83c3-f823-49f5-9862-6103e98a970a-config-out\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.277921 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.277898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.277921 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.277922    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.278342 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.277946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b6c83c3-f823-49f5-9862-6103e98a970a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.278342 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.277970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.278342 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.277999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.278342 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.278025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.278342 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.278075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-web-config\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.278342 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.278096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.278342 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.278121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.278342 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.278151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.278342 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.278179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8b6c83c3-f823-49f5-9862-6103e98a970a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.278816 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.278371    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.278816 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.278472    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.278816 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.278539    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8b6c83c3-f823-49f5-9862-6103e98a970a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.280710 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.279317    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.281301 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.280940    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-config\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.281301 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.280977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.281301 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.281056    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.281301 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.281189    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.281301 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.281204    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-web-config\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.281301 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.281235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b6c83c3-f823-49f5-9862-6103e98a970a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.281301 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.281255    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.281717 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.281615    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.281717 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.281692    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.282093 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.282074    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.283416 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.283397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.283497 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.283397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8b6c83c3-f823-49f5-9862-6103e98a970a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.283497 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.283452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b6c83c3-f823-49f5-9862-6103e98a970a-config-out\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.283497 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.283451    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.283896 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.283880    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8b6c83c3-f823-49f5-9862-6103e98a970a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.289119 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.289099    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc78g\" (UniqueName: \"kubernetes.io/projected/8b6c83c3-f823-49f5-9862-6103e98a970a-kube-api-access-hc78g\") pod \"prometheus-k8s-0\" (UID: \"8b6c83c3-f823-49f5-9862-6103e98a970a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.386375 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.386331    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:49.522142 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.522055    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:45:49.525706 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:45:49.525682    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6c83c3_f823_49f5_9862_6103e98a970a.slice/crio-c1f3f5c787a7ff6b8a10502aa066ec29307b2a403af0e10681765ecbadb22e76 WatchSource:0}: Error finding container c1f3f5c787a7ff6b8a10502aa066ec29307b2a403af0e10681765ecbadb22e76: Status 404 returned error can't find the container with id c1f3f5c787a7ff6b8a10502aa066ec29307b2a403af0e10681765ecbadb22e76
Jul 24 17:45:49.710759 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.710723    2578 generic.go:356] "Generic (PLEG): container finished" podID="8b6c83c3-f823-49f5-9862-6103e98a970a" containerID="4342e9f61cd34e6a811ee7bcf45cd77d761762c60eca5e69707f2cceed3e917b" exitCode=0
Jul 24 17:45:49.711158 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.710803    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b6c83c3-f823-49f5-9862-6103e98a970a","Type":"ContainerDied","Data":"4342e9f61cd34e6a811ee7bcf45cd77d761762c60eca5e69707f2cceed3e917b"}
Jul 24 17:45:49.711158 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.710871    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b6c83c3-f823-49f5-9862-6103e98a970a","Type":"ContainerStarted","Data":"c1f3f5c787a7ff6b8a10502aa066ec29307b2a403af0e10681765ecbadb22e76"}
Jul 24 17:45:49.968277 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:49.968250    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97bcd8e2-25db-4ef1-a41b-9584825b896b" path="/var/lib/kubelet/pods/97bcd8e2-25db-4ef1-a41b-9584825b896b/volumes"
Jul 24 17:45:50.717948 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:50.717913    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b6c83c3-f823-49f5-9862-6103e98a970a","Type":"ContainerStarted","Data":"3986f6e29a8c46b9e27289e024d10a2051a315ac8b24d4084bf4c5904ae63742"}
Jul 24 17:45:50.718334 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:50.717981    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b6c83c3-f823-49f5-9862-6103e98a970a","Type":"ContainerStarted","Data":"42e05e293d52decf86e0f45e9784ba444044a1db82f5ef3b7a87ca658412cca0"}
Jul 24 17:45:50.718334 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:50.718011    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b6c83c3-f823-49f5-9862-6103e98a970a","Type":"ContainerStarted","Data":"674f94785ea5a1140555fffc66d96250418da070400d57100c1fba64171c7d88"}
Jul 24 17:45:50.718334 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:50.718047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b6c83c3-f823-49f5-9862-6103e98a970a","Type":"ContainerStarted","Data":"7e6f1b067e8ab20af890a575992501f636a99a35b6fbd1e097a8497090356406"}
Jul 24 17:45:50.718334 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:50.718076    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b6c83c3-f823-49f5-9862-6103e98a970a","Type":"ContainerStarted","Data":"397a6c89e596a323b339aec0e77121100342f5ec3cd52bce899e0e1ae884370e"}
Jul 24 17:45:50.718334 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:50.718138    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b6c83c3-f823-49f5-9862-6103e98a970a","Type":"ContainerStarted","Data":"4360a7fd83e01bea85779c05b6c05da87bbd7db1573e4f8fc6c0a6d77632e631"}
Jul 24 17:45:50.750229 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:50.750168    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.736239783 podStartE2EDuration="1.750125436s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.885653ms" isStatefulPod=false podCreationTimestamp="2026-07-24 17:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:45:50.748096339 +0000 UTC m=+173.271454830" watchObservedRunningTime="2026-07-24 17:45:50.750125436 +0000 UTC m=+173.273483913"
Jul 24 17:45:50.821579 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:50.821547    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:50.901280 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:50.901248    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:52.425221 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:52.425191    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:52.525958 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:52.525928    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:54.386834 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:54.386770    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:58.879229 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:58.879191    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:58.978190 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:45:58.978153    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:20.666089 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:20.666060    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:20.765504 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:20.765468    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:37.408829 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:37.408770    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:42.992388 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:42.992355    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:43.104996 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:43.104967    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:46.982704 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:46.982666    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:47.082224 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:47.082193    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:48.841482 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:48.841452    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:48.984629 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:48.984600    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:49.261088 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:49.261007    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:49.387160 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:49.387101    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:46:49.402565 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:49.402541    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:46:49.462165 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:49.462137    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:49.953754 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:49.953720    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:46:50.223301 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:50.223226    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:50.323276 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:50.323249    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:51.519368 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:51.519334    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:51.935486 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:51.935455    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:53.168323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:53.168290    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:53.268988 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:53.268957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:59.706526 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:46:59.706495    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:01.189937 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:01.189896    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:01.288070 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:01.288022    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:06.272500 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:06.272441    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:06.373190 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:06.373162    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:08.520691 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:08.520657    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:19.847415 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:19.847345    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:19.950613 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:19.950586    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:23.782146 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:23.782109    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:23.873406 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:23.873377    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:24.402892 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.402864    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:24.591697 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.591632    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:24.692495 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.692378    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:24.720891 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.720861    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"]
Jul 24 17:47:24.724097 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.724077    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"
Jul 24 17:47:24.727216 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.727184    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 24 17:47:24.727407 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.727381    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vnvl4\""
Jul 24 17:47:24.727407 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.727232    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 17:47:24.727523 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.727261    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 17:47:24.727590 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.727568    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 17:47:24.753898 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.753873    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"]
Jul 24 17:47:24.848161 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.848123    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhs6r\" (UniqueName: \"kubernetes.io/projected/5a7ebe25-8c76-432e-ac51-7b8180d3e4bc-kube-api-access-mhs6r\") pod \"opendatahub-operator-controller-manager-dbf4bd545-bxczs\" (UID: \"5a7ebe25-8c76-432e-ac51-7b8180d3e4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"
Jul 24 17:47:24.848161 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.848164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a7ebe25-8c76-432e-ac51-7b8180d3e4bc-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbf4bd545-bxczs\" (UID: \"5a7ebe25-8c76-432e-ac51-7b8180d3e4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"
Jul 24 17:47:24.848571 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.848187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a7ebe25-8c76-432e-ac51-7b8180d3e4bc-tmp\") pod \"opendatahub-operator-controller-manager-dbf4bd545-bxczs\" (UID: \"5a7ebe25-8c76-432e-ac51-7b8180d3e4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"
Jul 24 17:47:24.848571 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.848238    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5a7ebe25-8c76-432e-ac51-7b8180d3e4bc-manifests\") pod \"opendatahub-operator-controller-manager-dbf4bd545-bxczs\" (UID: \"5a7ebe25-8c76-432e-ac51-7b8180d3e4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"
Jul 24 17:47:24.848571 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.848316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5a7ebe25-8c76-432e-ac51-7b8180d3e4bc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbf4bd545-bxczs\" (UID: \"5a7ebe25-8c76-432e-ac51-7b8180d3e4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"
Jul 24 17:47:24.935431 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.935401    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:24.948986 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.948914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a7ebe25-8c76-432e-ac51-7b8180d3e4bc-tmp\") pod \"opendatahub-operator-controller-manager-dbf4bd545-bxczs\" (UID: \"5a7ebe25-8c76-432e-ac51-7b8180d3e4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"
Jul 24 17:47:24.948986 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.948943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5a7ebe25-8c76-432e-ac51-7b8180d3e4bc-manifests\") pod \"opendatahub-operator-controller-manager-dbf4bd545-bxczs\" (UID: \"5a7ebe25-8c76-432e-ac51-7b8180d3e4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"
Jul 24 17:47:24.948986 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.948981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5a7ebe25-8c76-432e-ac51-7b8180d3e4bc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbf4bd545-bxczs\" (UID: \"5a7ebe25-8c76-432e-ac51-7b8180d3e4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"
Jul 24 17:47:24.949232 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.949019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhs6r\" (UniqueName: \"kubernetes.io/projected/5a7ebe25-8c76-432e-ac51-7b8180d3e4bc-kube-api-access-mhs6r\") pod \"opendatahub-operator-controller-manager-dbf4bd545-bxczs\" (UID: \"5a7ebe25-8c76-432e-ac51-7b8180d3e4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"
Jul 24 17:47:24.949232 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.949042    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a7ebe25-8c76-432e-ac51-7b8180d3e4bc-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbf4bd545-bxczs\" (UID: \"5a7ebe25-8c76-432e-ac51-7b8180d3e4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"
Jul 24 17:47:24.949504 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.949473    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5a7ebe25-8c76-432e-ac51-7b8180d3e4bc-manifests\") pod \"opendatahub-operator-controller-manager-dbf4bd545-bxczs\" (UID: \"5a7ebe25-8c76-432e-ac51-7b8180d3e4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"
Jul 24 17:47:24.951080 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.951057    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a7ebe25-8c76-432e-ac51-7b8180d3e4bc-tmp\") pod \"opendatahub-operator-controller-manager-dbf4bd545-bxczs\" (UID: \"5a7ebe25-8c76-432e-ac51-7b8180d3e4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"
Jul 24 17:47:24.951407 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.951385    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5a7ebe25-8c76-432e-ac51-7b8180d3e4bc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbf4bd545-bxczs\" (UID: \"5a7ebe25-8c76-432e-ac51-7b8180d3e4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"
Jul 24 17:47:24.951454 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.951385    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a7ebe25-8c76-432e-ac51-7b8180d3e4bc-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbf4bd545-bxczs\" (UID: \"5a7ebe25-8c76-432e-ac51-7b8180d3e4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"
Jul 24 17:47:24.956614 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:24.956592    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhs6r\" (UniqueName: \"kubernetes.io/projected/5a7ebe25-8c76-432e-ac51-7b8180d3e4bc-kube-api-access-mhs6r\") pod \"opendatahub-operator-controller-manager-dbf4bd545-bxczs\" (UID: \"5a7ebe25-8c76-432e-ac51-7b8180d3e4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"
Jul 24 17:47:25.032707 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:25.032673    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:25.035566 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:25.035546    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"
Jul 24 17:47:25.133463 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:25.133434    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:25.165547 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:25.165519    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"]
Jul 24 17:47:25.168160 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:47:25.168124    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7ebe25_8c76_432e_ac51_7b8180d3e4bc.slice/crio-28e1ecb2a35c60112a7e1bd293761fc80f0acfda7720c002b3e1e5d709a78091 WatchSource:0}: Error finding container 28e1ecb2a35c60112a7e1bd293761fc80f0acfda7720c002b3e1e5d709a78091: Status 404 returned error can't find the container with id 28e1ecb2a35c60112a7e1bd293761fc80f0acfda7720c002b3e1e5d709a78091
Jul 24 17:47:25.210840 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:25.210755    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:25.270451 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:25.270429    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:25.370682 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:25.370656    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:25.469533 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:25.469452    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:25.575191 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:25.575163    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:25.670506 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:25.670475    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:26.069742 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:26.069703    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs" event={"ID":"5a7ebe25-8c76-432e-ac51-7b8180d3e4bc","Type":"ContainerStarted","Data":"28e1ecb2a35c60112a7e1bd293761fc80f0acfda7720c002b3e1e5d709a78091"}
Jul 24 17:47:28.735656 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:28.735624    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:28.848064 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:28.848026    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:29.084708 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:29.084668    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs" event={"ID":"5a7ebe25-8c76-432e-ac51-7b8180d3e4bc","Type":"ContainerStarted","Data":"38f503198c9cfbd4dc5639a097b03d6e559a9bcb19691d46ca222806ac233064"}
Jul 24 17:47:30.089832 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:30.089800    2578 generic.go:356] "Generic (PLEG): container finished" podID="5a7ebe25-8c76-432e-ac51-7b8180d3e4bc" containerID="38f503198c9cfbd4dc5639a097b03d6e559a9bcb19691d46ca222806ac233064" exitCode=0
Jul 24 17:47:30.090228 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:30.089845    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs" event={"ID":"5a7ebe25-8c76-432e-ac51-7b8180d3e4bc","Type":"ContainerDied","Data":"38f503198c9cfbd4dc5639a097b03d6e559a9bcb19691d46ca222806ac233064"}
Jul 24 17:47:31.095765 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:31.095733    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs" event={"ID":"5a7ebe25-8c76-432e-ac51-7b8180d3e4bc","Type":"ContainerStarted","Data":"c11208d669e45ce792dd94a45b3d16cc68c5cf1acdd70c0ec030def759bcbb9a"}
Jul 24 17:47:31.096202 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:31.095966    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"
Jul 24 17:47:31.130773 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:31.130723    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs" podStartSLOduration=2.712475339 podStartE2EDuration="7.13070722s" totalImagesPullingTime="3.269961503s" totalInitContainerRuntime="1.148270378s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:47:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:47:31.128849253 +0000 UTC m=+273.652207739" watchObservedRunningTime="2026-07-24 17:47:31.13070722 +0000 UTC m=+273.654065764"
Jul 24 17:47:42.102107 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:42.102070    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-bxczs"
Jul 24 17:47:43.906768 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:43.906735    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:44.616281 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.616245    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-56c646dc89-5bnkz"]
Jul 24 17:47:44.625030 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.625011    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-56c646dc89-5bnkz"
Jul 24 17:47:44.627830 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.627803    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 24 17:47:44.627954 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.627869    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 24 17:47:44.627954 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.627895    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 17:47:44.628871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.628852    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xcmzs\""
Jul 24 17:47:44.628928 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.628870    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 17:47:44.635522 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.635432    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-56c646dc89-5bnkz"]
Jul 24 17:47:44.711571 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.711542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97e8e45e-ee3a-44bf-933e-78c64cf63e36-tmp\") pod \"kube-auth-proxy-56c646dc89-5bnkz\" (UID: \"97e8e45e-ee3a-44bf-933e-78c64cf63e36\") " pod="openshift-ingress/kube-auth-proxy-56c646dc89-5bnkz"
Jul 24 17:47:44.711727 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.711623    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97e8e45e-ee3a-44bf-933e-78c64cf63e36-tls-certs\") pod \"kube-auth-proxy-56c646dc89-5bnkz\" (UID: \"97e8e45e-ee3a-44bf-933e-78c64cf63e36\") " pod="openshift-ingress/kube-auth-proxy-56c646dc89-5bnkz"
Jul 24 17:47:44.711727 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.711651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5g8d\" (UniqueName: \"kubernetes.io/projected/97e8e45e-ee3a-44bf-933e-78c64cf63e36-kube-api-access-k5g8d\") pod \"kube-auth-proxy-56c646dc89-5bnkz\" (UID: \"97e8e45e-ee3a-44bf-933e-78c64cf63e36\") " pod="openshift-ingress/kube-auth-proxy-56c646dc89-5bnkz"
Jul 24 17:47:44.714251 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.714225    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-265d5"]
Jul 24 17:47:44.716296 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.716279    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-265d5"
Jul 24 17:47:44.718981 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.718964    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2px7t\""
Jul 24 17:47:44.719348 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.719331    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 24 17:47:44.737894 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.737863    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-265d5"]
Jul 24 17:47:44.812907 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.812873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgk9s\" (UniqueName: \"kubernetes.io/projected/35cb65cd-f8ec-4f02-a799-6c0f67c254a7-kube-api-access-kgk9s\") pod \"odh-model-controller-7998c67dc4-265d5\" (UID: \"35cb65cd-f8ec-4f02-a799-6c0f67c254a7\") " pod="opendatahub/odh-model-controller-7998c67dc4-265d5"
Jul 24 17:47:44.813096 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.812925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97e8e45e-ee3a-44bf-933e-78c64cf63e36-tls-certs\") pod \"kube-auth-proxy-56c646dc89-5bnkz\" (UID: \"97e8e45e-ee3a-44bf-933e-78c64cf63e36\") " pod="openshift-ingress/kube-auth-proxy-56c646dc89-5bnkz"
Jul 24 17:47:44.813096 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.812972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35cb65cd-f8ec-4f02-a799-6c0f67c254a7-cert\") pod \"odh-model-controller-7998c67dc4-265d5\" (UID: \"35cb65cd-f8ec-4f02-a799-6c0f67c254a7\") " pod="opendatahub/odh-model-controller-7998c67dc4-265d5"
Jul 24 17:47:44.813096 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.813011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5g8d\" (UniqueName: \"kubernetes.io/projected/97e8e45e-ee3a-44bf-933e-78c64cf63e36-kube-api-access-k5g8d\") pod \"kube-auth-proxy-56c646dc89-5bnkz\" (UID: \"97e8e45e-ee3a-44bf-933e-78c64cf63e36\") " pod="openshift-ingress/kube-auth-proxy-56c646dc89-5bnkz"
Jul 24 17:47:44.813096 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.813082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97e8e45e-ee3a-44bf-933e-78c64cf63e36-tmp\") pod \"kube-auth-proxy-56c646dc89-5bnkz\" (UID: \"97e8e45e-ee3a-44bf-933e-78c64cf63e36\") " pod="openshift-ingress/kube-auth-proxy-56c646dc89-5bnkz"
Jul 24 17:47:44.815145 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.815125    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97e8e45e-ee3a-44bf-933e-78c64cf63e36-tmp\") pod \"kube-auth-proxy-56c646dc89-5bnkz\" (UID: \"97e8e45e-ee3a-44bf-933e-78c64cf63e36\") " pod="openshift-ingress/kube-auth-proxy-56c646dc89-5bnkz"
Jul 24 17:47:44.815282 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.815266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97e8e45e-ee3a-44bf-933e-78c64cf63e36-tls-certs\") pod \"kube-auth-proxy-56c646dc89-5bnkz\" (UID: \"97e8e45e-ee3a-44bf-933e-78c64cf63e36\") " pod="openshift-ingress/kube-auth-proxy-56c646dc89-5bnkz"
Jul 24 17:47:44.829456 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.829438    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5g8d\" (UniqueName: \"kubernetes.io/projected/97e8e45e-ee3a-44bf-933e-78c64cf63e36-kube-api-access-k5g8d\") pod \"kube-auth-proxy-56c646dc89-5bnkz\" (UID: \"97e8e45e-ee3a-44bf-933e-78c64cf63e36\") " pod="openshift-ingress/kube-auth-proxy-56c646dc89-5bnkz"
Jul 24 17:47:44.914061 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.913981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgk9s\" (UniqueName: \"kubernetes.io/projected/35cb65cd-f8ec-4f02-a799-6c0f67c254a7-kube-api-access-kgk9s\") pod \"odh-model-controller-7998c67dc4-265d5\" (UID: \"35cb65cd-f8ec-4f02-a799-6c0f67c254a7\") " pod="opendatahub/odh-model-controller-7998c67dc4-265d5"
Jul 24 17:47:44.914061 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.914035    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35cb65cd-f8ec-4f02-a799-6c0f67c254a7-cert\") pod \"odh-model-controller-7998c67dc4-265d5\" (UID: \"35cb65cd-f8ec-4f02-a799-6c0f67c254a7\") " pod="opendatahub/odh-model-controller-7998c67dc4-265d5"
Jul 24 17:47:44.914437 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:47:44.914135    2578 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 24 17:47:44.914437 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:47:44.914208    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35cb65cd-f8ec-4f02-a799-6c0f67c254a7-cert podName:35cb65cd-f8ec-4f02-a799-6c0f67c254a7 nodeName:}" failed. No retries permitted until 2026-07-24 17:47:45.414186467 +0000 UTC m=+287.937544936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/35cb65cd-f8ec-4f02-a799-6c0f67c254a7-cert") pod "odh-model-controller-7998c67dc4-265d5" (UID: "35cb65cd-f8ec-4f02-a799-6c0f67c254a7") : secret "odh-model-controller-webhook-cert" not found
Jul 24 17:47:44.928814 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.928769    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgk9s\" (UniqueName: \"kubernetes.io/projected/35cb65cd-f8ec-4f02-a799-6c0f67c254a7-kube-api-access-kgk9s\") pod \"odh-model-controller-7998c67dc4-265d5\" (UID: \"35cb65cd-f8ec-4f02-a799-6c0f67c254a7\") " pod="opendatahub/odh-model-controller-7998c67dc4-265d5"
Jul 24 17:47:44.934349 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:44.934329    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-56c646dc89-5bnkz"
Jul 24 17:47:45.067747 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:45.067580    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-56c646dc89-5bnkz"]
Jul 24 17:47:45.070300 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:47:45.070276    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e8e45e_ee3a_44bf_933e_78c64cf63e36.slice/crio-f312209636ff2ec66046ec34d03ce6064daa1e6820601d4999588e18995c4e37 WatchSource:0}: Error finding container f312209636ff2ec66046ec34d03ce6064daa1e6820601d4999588e18995c4e37: Status 404 returned error can't find the container with id f312209636ff2ec66046ec34d03ce6064daa1e6820601d4999588e18995c4e37
Jul 24 17:47:45.150567 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:45.150539    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-56c646dc89-5bnkz" event={"ID":"97e8e45e-ee3a-44bf-933e-78c64cf63e36","Type":"ContainerStarted","Data":"f312209636ff2ec66046ec34d03ce6064daa1e6820601d4999588e18995c4e37"}
Jul 24 17:47:45.152351 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:45.152324    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:45.419233 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:45.419200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35cb65cd-f8ec-4f02-a799-6c0f67c254a7-cert\") pod \"odh-model-controller-7998c67dc4-265d5\" (UID: \"35cb65cd-f8ec-4f02-a799-6c0f67c254a7\") " pod="opendatahub/odh-model-controller-7998c67dc4-265d5"
Jul 24 17:47:45.421508 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:45.421481    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35cb65cd-f8ec-4f02-a799-6c0f67c254a7-cert\") pod \"odh-model-controller-7998c67dc4-265d5\" (UID: \"35cb65cd-f8ec-4f02-a799-6c0f67c254a7\") " pod="opendatahub/odh-model-controller-7998c67dc4-265d5"
Jul 24 17:47:45.625790 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:45.625752    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-265d5"
Jul 24 17:47:45.785264 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:45.785236    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-265d5"]
Jul 24 17:47:45.787622 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:47:45.787590    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cb65cd_f8ec_4f02_a799_6c0f67c254a7.slice/crio-a13d9d51408e0a41aa237921997884020e4d7f5754ac5d304b6adb6502c4af89 WatchSource:0}: Error finding container a13d9d51408e0a41aa237921997884020e4d7f5754ac5d304b6adb6502c4af89: Status 404 returned error can't find the container with id a13d9d51408e0a41aa237921997884020e4d7f5754ac5d304b6adb6502c4af89
Jul 24 17:47:45.812600 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:45.812576    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:45.950904 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:45.950819    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:45.952627 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:45.952608    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"]
Jul 24 17:47:45.955222 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:45.955208    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:45.957638 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:45.957619    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 24 17:47:45.957756 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:45.957691    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 24 17:47:45.957910 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:45.957896    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 24 17:47:45.958109 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:45.958095    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 24 17:47:45.958860 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:45.958841    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rgbr6\""
Jul 24 17:47:45.970300 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:45.970280    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"]
Jul 24 17:47:46.031600 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.031569    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:46.124671 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.124642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.124871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.124676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.124871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.124713    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.124871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.124736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.124871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.124798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwhgw\" (UniqueName: \"kubernetes.io/projected/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-kube-api-access-qwhgw\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.125100 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.124881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.125100 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.124919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.160111 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.160082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-265d5" event={"ID":"35cb65cd-f8ec-4f02-a799-6c0f67c254a7","Type":"ContainerStarted","Data":"a13d9d51408e0a41aa237921997884020e4d7f5754ac5d304b6adb6502c4af89"}
Jul 24 17:47:46.214144 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.214065    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:46.226248 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.226216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.226380 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.226260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.226380 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.226317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.226380 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.226345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.226646 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.226378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.226739 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.226681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.226827 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.226736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwhgw\" (UniqueName: \"kubernetes.io/projected/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-kube-api-access-qwhgw\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.226993 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.226943    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.229295 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.229263    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.229395 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.229300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.229518 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.229495    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.229570 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.229502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.236053 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.236029    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.236218 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.236197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwhgw\" (UniqueName: \"kubernetes.io/projected/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-kube-api-access-qwhgw\") pod \"istiod-openshift-gateway-7b7995d5cd-xmjlq\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.265694 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.265670    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:46.423383 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:46.423352    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"]
Jul 24 17:47:46.425658 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:47:46.425615    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af4d66b_a09e_48bf_85b0_c844e80c2c1f.slice/crio-ecb9c8135504aad0e8cce13c4e2b3d66cd2ded976604dae87267da4d5edbacce WatchSource:0}: Error finding container ecb9c8135504aad0e8cce13c4e2b3d66cd2ded976604dae87267da4d5edbacce: Status 404 returned error can't find the container with id ecb9c8135504aad0e8cce13c4e2b3d66cd2ded976604dae87267da4d5edbacce
Jul 24 17:47:47.168990 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:47.168919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq" event={"ID":"4af4d66b-a09e-48bf-85b0-c844e80c2c1f","Type":"ContainerStarted","Data":"ecb9c8135504aad0e8cce13c4e2b3d66cd2ded976604dae87267da4d5edbacce"}
Jul 24 17:47:47.232729 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:47.232641    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:50.325455 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:50.325423    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 17:47:50.325722 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:50.325485    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 17:47:50.943404 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:50.943371    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:51.037546 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:51.037516    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:51.187698 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:51.187663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-56c646dc89-5bnkz" event={"ID":"97e8e45e-ee3a-44bf-933e-78c64cf63e36","Type":"ContainerStarted","Data":"1130c22186a57ad5a85451262453ca8f89fafb0a74147045e8f260578f1cb04e"}
Jul 24 17:47:51.195256 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:51.195168    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq" event={"ID":"4af4d66b-a09e-48bf-85b0-c844e80c2c1f","Type":"ContainerStarted","Data":"91866cf7a46c70c01122328483e7e108c5c9d49fa9f1c5a62642b692ba95e005"}
Jul 24 17:47:51.195539 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:51.195515    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:51.197306 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:51.197276    2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-xmjlq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 17:47:51.197439 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:51.197329    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq" podUID="4af4d66b-a09e-48bf-85b0-c844e80c2c1f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 17:47:51.197986 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:51.197958    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-265d5" event={"ID":"35cb65cd-f8ec-4f02-a799-6c0f67c254a7","Type":"ContainerStarted","Data":"fd17726a4f47f9e1d660c09a1a93caafeadfcb68a0dea1a7d1189652a155d239"}
Jul 24 17:47:51.198126 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:51.198107    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-265d5"
Jul 24 17:47:51.212522 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:51.212472    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-56c646dc89-5bnkz" podStartSLOduration=1.965109981 podStartE2EDuration="7.212456197s" totalImagesPullingTime="5.247346216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:47:51.211082088 +0000 UTC m=+293.734440584" watchObservedRunningTime="2026-07-24 17:47:51.212456197 +0000 UTC m=+293.735815018"
Jul 24 17:47:51.222150 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:51.222121    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:51.235900 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:51.235842    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq" podStartSLOduration=2.34075985 podStartE2EDuration="6.235825407s" totalImagesPullingTime="3.895065557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:47:51.234557779 +0000 UTC m=+293.757916258" watchObservedRunningTime="2026-07-24 17:47:51.235825407 +0000 UTC m=+293.759183875"
Jul 24 17:47:51.256635 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:51.256583    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-265d5" podStartSLOduration=2.671203643 podStartE2EDuration="7.256561978s" totalImagesPullingTime="4.585358335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:47:51.255017394 +0000 UTC m=+293.778375899" watchObservedRunningTime="2026-07-24 17:47:51.256561978 +0000 UTC m=+293.779920456"
Jul 24 17:47:51.468183 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:51.468101    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:51.567577 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:51.567520    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:52.202724 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:52.202699    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:54.143504 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.143456    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"]
Jul 24 17:47:54.146694 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.146676    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.170195 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.170164    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"]
Jul 24 17:47:54.305133 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.305096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7913e838-8015-4554-aa79-6fa18a9d537b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.305133 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.305136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7913e838-8015-4554-aa79-6fa18a9d537b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.305333 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.305167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7913e838-8015-4554-aa79-6fa18a9d537b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.305333 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.305220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7913e838-8015-4554-aa79-6fa18a9d537b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.305333 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.305278    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7913e838-8015-4554-aa79-6fa18a9d537b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.305333 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.305307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7913e838-8015-4554-aa79-6fa18a9d537b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.305333 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.305330    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh2jh\" (UniqueName: \"kubernetes.io/projected/7913e838-8015-4554-aa79-6fa18a9d537b-kube-api-access-sh2jh\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.406017 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.405936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7913e838-8015-4554-aa79-6fa18a9d537b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.406017 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.405984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7913e838-8015-4554-aa79-6fa18a9d537b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.406017 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.406007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7913e838-8015-4554-aa79-6fa18a9d537b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.406305 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.406031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh2jh\" (UniqueName: \"kubernetes.io/projected/7913e838-8015-4554-aa79-6fa18a9d537b-kube-api-access-sh2jh\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.406305 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.406081    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7913e838-8015-4554-aa79-6fa18a9d537b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.406305 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.406110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7913e838-8015-4554-aa79-6fa18a9d537b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.406305 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.406154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7913e838-8015-4554-aa79-6fa18a9d537b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.406712 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.406687    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7913e838-8015-4554-aa79-6fa18a9d537b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.408272 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.408252    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7913e838-8015-4554-aa79-6fa18a9d537b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.408563 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.408541    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7913e838-8015-4554-aa79-6fa18a9d537b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.408763 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.408748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7913e838-8015-4554-aa79-6fa18a9d537b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.408844 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.408772    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7913e838-8015-4554-aa79-6fa18a9d537b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.420086 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.420064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh2jh\" (UniqueName: \"kubernetes.io/projected/7913e838-8015-4554-aa79-6fa18a9d537b-kube-api-access-sh2jh\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.423671 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.423649    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7913e838-8015-4554-aa79-6fa18a9d537b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xxd6h\" (UID: \"7913e838-8015-4554-aa79-6fa18a9d537b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.455766 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.455740    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:54.611424 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.611391    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"]
Jul 24 17:47:54.614197 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:47:54.614171    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7913e838_8015_4554_aa79_6fa18a9d537b.slice/crio-0bf7f517d401f0c040c31812f1b11b1a4a52f21be59eecca8e8d2c502daa1061 WatchSource:0}: Error finding container 0bf7f517d401f0c040c31812f1b11b1a4a52f21be59eecca8e8d2c502daa1061: Status 404 returned error can't find the container with id 0bf7f517d401f0c040c31812f1b11b1a4a52f21be59eecca8e8d2c502daa1061
Jul 24 17:47:54.617254 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.617227    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 17:47:54.617315 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:54.617282    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 17:47:55.222985 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:55.222943    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h" event={"ID":"7913e838-8015-4554-aa79-6fa18a9d537b","Type":"ContainerStarted","Data":"4e078ed1258bd0f49b8bd89dc1ab3c3ad1c9c974b50eb5cf7642154d17d27cdd"}
Jul 24 17:47:55.223450 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:55.223044    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h" event={"ID":"7913e838-8015-4554-aa79-6fa18a9d537b","Type":"ContainerStarted","Data":"0bf7f517d401f0c040c31812f1b11b1a4a52f21be59eecca8e8d2c502daa1061"}
Jul 24 17:47:55.224361 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:55.224333    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:55.225608 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:55.225577    2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-xxd6h container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 17:47:55.225710 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:55.225621    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h" podUID="7913e838-8015-4554-aa79-6fa18a9d537b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 17:47:55.266135 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:55.266076    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h" podStartSLOduration=1.266054449 podStartE2EDuration="1.266054449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:47:55.265158047 +0000 UTC m=+297.788516524" watchObservedRunningTime="2026-07-24 17:47:55.266054449 +0000 UTC m=+297.789412929"
Jul 24 17:47:56.228343 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.228314    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xxd6h"
Jul 24 17:47:56.328798 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.328743    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:56.336615 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.336583    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"]
Jul 24 17:47:56.336809 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.336750    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq" podUID="4af4d66b-a09e-48bf-85b0-c844e80c2c1f" containerName="discovery" containerID="cri-o://91866cf7a46c70c01122328483e7e108c5c9d49fa9f1c5a62642b692ba95e005" gracePeriod=30
Jul 24 17:47:56.468116 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.468087    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:56.547826 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.547794    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:56.585453 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.585425    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:56.634836 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.634801    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-csr-ca-configmap\") pod \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") "
Jul 24 17:47:56.635008 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.634926    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-csr-dns-cert\") pod \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") "
Jul 24 17:47:56.635008 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.634981    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-cacerts\") pod \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") "
Jul 24 17:47:56.635136 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.635020    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-local-certs\") pod \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") "
Jul 24 17:47:56.635136 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.635068    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-token\") pod \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") "
Jul 24 17:47:56.635136 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.635121    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "4af4d66b-a09e-48bf-85b0-c844e80c2c1f" (UID: "4af4d66b-a09e-48bf-85b0-c844e80c2c1f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:47:56.635291 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.635177    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-kubeconfig\") pod \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") "
Jul 24 17:47:56.635291 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.635248    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwhgw\" (UniqueName: \"kubernetes.io/projected/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-kube-api-access-qwhgw\") pod \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\" (UID: \"4af4d66b-a09e-48bf-85b0-c844e80c2c1f\") "
Jul 24 17:47:56.635544 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.635522    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-csr-ca-configmap\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:47:56.637273 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.637239    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-local-certs" (OuterVolumeSpecName: "local-certs") pod "4af4d66b-a09e-48bf-85b0-c844e80c2c1f" (UID: "4af4d66b-a09e-48bf-85b0-c844e80c2c1f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:47:56.637411 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.637306    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-token" (OuterVolumeSpecName: "istio-token") pod "4af4d66b-a09e-48bf-85b0-c844e80c2c1f" (UID: "4af4d66b-a09e-48bf-85b0-c844e80c2c1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:47:56.637487 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.637468    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-cacerts" (OuterVolumeSpecName: "cacerts") pod "4af4d66b-a09e-48bf-85b0-c844e80c2c1f" (UID: "4af4d66b-a09e-48bf-85b0-c844e80c2c1f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:47:56.637660 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.637639    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "4af4d66b-a09e-48bf-85b0-c844e80c2c1f" (UID: "4af4d66b-a09e-48bf-85b0-c844e80c2c1f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:47:56.637722 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.637663    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-kube-api-access-qwhgw" (OuterVolumeSpecName: "kube-api-access-qwhgw") pod "4af4d66b-a09e-48bf-85b0-c844e80c2c1f" (UID: "4af4d66b-a09e-48bf-85b0-c844e80c2c1f"). InnerVolumeSpecName "kube-api-access-qwhgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:47:56.637722 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.637667    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "4af4d66b-a09e-48bf-85b0-c844e80c2c1f" (UID: "4af4d66b-a09e-48bf-85b0-c844e80c2c1f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:47:56.736867 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.736825    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-csr-dns-cert\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:47:56.736867 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.736860    2578 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-cacerts\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:47:56.736867 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.736872    2578 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-local-certs\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:47:56.737114 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.736885    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-token\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:47:56.737114 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.736896    2578 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-istio-kubeconfig\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:47:56.737114 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:56.736907    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwhgw\" (UniqueName: \"kubernetes.io/projected/4af4d66b-a09e-48bf-85b0-c844e80c2c1f-kube-api-access-qwhgw\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:47:57.188697 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:57.188664    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:57.233510 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:57.233479    2578 generic.go:356] "Generic (PLEG): container finished" podID="4af4d66b-a09e-48bf-85b0-c844e80c2c1f" containerID="91866cf7a46c70c01122328483e7e108c5c9d49fa9f1c5a62642b692ba95e005" exitCode=0
Jul 24 17:47:57.233998 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:57.233536    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"
Jul 24 17:47:57.233998 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:57.233566    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq" event={"ID":"4af4d66b-a09e-48bf-85b0-c844e80c2c1f","Type":"ContainerDied","Data":"91866cf7a46c70c01122328483e7e108c5c9d49fa9f1c5a62642b692ba95e005"}
Jul 24 17:47:57.233998 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:57.233614    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq" event={"ID":"4af4d66b-a09e-48bf-85b0-c844e80c2c1f","Type":"ContainerDied","Data":"ecb9c8135504aad0e8cce13c4e2b3d66cd2ded976604dae87267da4d5edbacce"}
Jul 24 17:47:57.233998 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:57.233633    2578 scope.go:117] "RemoveContainer" containerID="91866cf7a46c70c01122328483e7e108c5c9d49fa9f1c5a62642b692ba95e005"
Jul 24 17:47:57.242261 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:57.242244    2578 scope.go:117] "RemoveContainer" containerID="91866cf7a46c70c01122328483e7e108c5c9d49fa9f1c5a62642b692ba95e005"
Jul 24 17:47:57.242523 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:47:57.242506    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91866cf7a46c70c01122328483e7e108c5c9d49fa9f1c5a62642b692ba95e005\": container with ID starting with 91866cf7a46c70c01122328483e7e108c5c9d49fa9f1c5a62642b692ba95e005 not found: ID does not exist" containerID="91866cf7a46c70c01122328483e7e108c5c9d49fa9f1c5a62642b692ba95e005"
Jul 24 17:47:57.242576 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:57.242534    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91866cf7a46c70c01122328483e7e108c5c9d49fa9f1c5a62642b692ba95e005"} err="failed to get container status \"91866cf7a46c70c01122328483e7e108c5c9d49fa9f1c5a62642b692ba95e005\": rpc error: code = NotFound desc = could not find container \"91866cf7a46c70c01122328483e7e108c5c9d49fa9f1c5a62642b692ba95e005\": container with ID starting with 91866cf7a46c70c01122328483e7e108c5c9d49fa9f1c5a62642b692ba95e005 not found: ID does not exist"
Jul 24 17:47:57.256323 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:57.256297    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"]
Jul 24 17:47:57.263391 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:57.263369    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmjlq"]
Jul 24 17:47:57.698793 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:57.698756    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:57.815828 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:57.815766    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:57.837338 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:57.836808    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 17:47:57.966793 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:47:57.966607    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4af4d66b-a09e-48bf-85b0-c844e80c2c1f" path="/var/lib/kubelet/pods/4af4d66b-a09e-48bf-85b0-c844e80c2c1f/volumes"
Jul 24 17:48:02.203723 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:48:02.203686    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-265d5"
Jul 24 17:48:18.904923 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:48:18.904885    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:45.761031 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:48:45.760999    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:45.856150 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:48:45.856119    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:52.897557 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:48:52.897467    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:52.955509 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:48:52.955475    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:58.859693 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:48:58.859652    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:00.936426 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:00.936393    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt"]
Jul 24 17:49:00.936825 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:00.936705    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af4d66b-a09e-48bf-85b0-c844e80c2c1f" containerName="discovery"
Jul 24 17:49:00.936825 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:00.936715    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af4d66b-a09e-48bf-85b0-c844e80c2c1f" containerName="discovery"
Jul 24 17:49:00.936825 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:00.936764    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4af4d66b-a09e-48bf-85b0-c844e80c2c1f" containerName="discovery"
Jul 24 17:49:00.938658 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:00.938643    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt"
Jul 24 17:49:00.942405 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:00.942383    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 17:49:00.943350 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:00.943335    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dvhpn\""
Jul 24 17:49:00.943425 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:00.943396    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 17:49:00.969283 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:00.969257    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt"]
Jul 24 17:49:00.984775 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:00.984749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrqvb\" (UniqueName: \"kubernetes.io/projected/c36f646e-eb2b-4f6e-9003-1010bd8418ca-kube-api-access-wrqvb\") pod \"limitador-operator-controller-manager-6c549b6d85-zvbpt\" (UID: \"c36f646e-eb2b-4f6e-9003-1010bd8418ca\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt"
Jul 24 17:49:01.085456 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:01.085420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrqvb\" (UniqueName: \"kubernetes.io/projected/c36f646e-eb2b-4f6e-9003-1010bd8418ca-kube-api-access-wrqvb\") pod \"limitador-operator-controller-manager-6c549b6d85-zvbpt\" (UID: \"c36f646e-eb2b-4f6e-9003-1010bd8418ca\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt"
Jul 24 17:49:01.102280 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:01.102249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrqvb\" (UniqueName: \"kubernetes.io/projected/c36f646e-eb2b-4f6e-9003-1010bd8418ca-kube-api-access-wrqvb\") pod \"limitador-operator-controller-manager-6c549b6d85-zvbpt\" (UID: \"c36f646e-eb2b-4f6e-9003-1010bd8418ca\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt"
Jul 24 17:49:01.248536 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:01.248458    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt"
Jul 24 17:49:01.390107 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:01.390073    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt"]
Jul 24 17:49:01.392489 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:49:01.392461    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36f646e_eb2b_4f6e_9003_1010bd8418ca.slice/crio-e33178585debef3e8f255dbb5f67fd1af48ca4471ec8556f38bbccab02336fb2 WatchSource:0}: Error finding container e33178585debef3e8f255dbb5f67fd1af48ca4471ec8556f38bbccab02336fb2: Status 404 returned error can't find the container with id e33178585debef3e8f255dbb5f67fd1af48ca4471ec8556f38bbccab02336fb2
Jul 24 17:49:01.394581 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:01.394563    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 17:49:01.500383 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:01.500304    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt" event={"ID":"c36f646e-eb2b-4f6e-9003-1010bd8418ca","Type":"ContainerStarted","Data":"e33178585debef3e8f255dbb5f67fd1af48ca4471ec8556f38bbccab02336fb2"}
Jul 24 17:49:01.674638 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:01.674591    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:03.081512 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:03.081475    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-5tlqg"]
Jul 24 17:49:03.087511 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:03.087486    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5tlqg"
Jul 24 17:49:03.091177 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:03.091152    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 24 17:49:03.092792 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:03.092752    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-bs68j\""
Jul 24 17:49:03.112969 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:03.112946    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-5tlqg"]
Jul 24 17:49:03.205503 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:03.205471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk8ff\" (UniqueName: \"kubernetes.io/projected/4d0cf1b5-e21d-4eca-bacc-5fc111b0549a-kube-api-access-kk8ff\") pod \"dns-operator-controller-manager-5985564776-5tlqg\" (UID: \"4d0cf1b5-e21d-4eca-bacc-5fc111b0549a\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-5tlqg"
Jul 24 17:49:03.253794 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:03.253755    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:03.306518 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:03.306487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk8ff\" (UniqueName: \"kubernetes.io/projected/4d0cf1b5-e21d-4eca-bacc-5fc111b0549a-kube-api-access-kk8ff\") pod \"dns-operator-controller-manager-5985564776-5tlqg\" (UID: \"4d0cf1b5-e21d-4eca-bacc-5fc111b0549a\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-5tlqg"
Jul 24 17:49:03.315634 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:03.315610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk8ff\" (UniqueName: \"kubernetes.io/projected/4d0cf1b5-e21d-4eca-bacc-5fc111b0549a-kube-api-access-kk8ff\") pod \"dns-operator-controller-manager-5985564776-5tlqg\" (UID: \"4d0cf1b5-e21d-4eca-bacc-5fc111b0549a\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-5tlqg"
Jul 24 17:49:03.399306 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:03.399223    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5tlqg"
Jul 24 17:49:03.516353 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:03.516288    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:03.657322 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:03.657241    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:03.765276 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:03.765246    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:03.887941 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:03.887924    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-5tlqg"]
Jul 24 17:49:03.889958 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:49:03.889928    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0cf1b5_e21d_4eca_bacc_5fc111b0549a.slice/crio-5432831266c8587d9612c335d5891c1558c1d57e32d0f9190b97f271538bc1f2 WatchSource:0}: Error finding container 5432831266c8587d9612c335d5891c1558c1d57e32d0f9190b97f271538bc1f2: Status 404 returned error can't find the container with id 5432831266c8587d9612c335d5891c1558c1d57e32d0f9190b97f271538bc1f2
Jul 24 17:49:04.515802 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:04.515759    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5tlqg" event={"ID":"4d0cf1b5-e21d-4eca-bacc-5fc111b0549a","Type":"ContainerStarted","Data":"5432831266c8587d9612c335d5891c1558c1d57e32d0f9190b97f271538bc1f2"}
Jul 24 17:49:04.517278 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:04.517249    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt" event={"ID":"c36f646e-eb2b-4f6e-9003-1010bd8418ca","Type":"ContainerStarted","Data":"0460454553a68b60a1fe4d63222899796f1b717b261f925acafeef4182785322"}
Jul 24 17:49:04.517405 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:04.517388    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt"
Jul 24 17:49:04.536820 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:04.536757    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt" podStartSLOduration=2.145572076 podStartE2EDuration="4.536741944s" totalImagesPullingTime="2.391169868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:49:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:49:04.535173426 +0000 UTC m=+367.058531901" watchObservedRunningTime="2026-07-24 17:49:04.536741944 +0000 UTC m=+367.060100421"
Jul 24 17:49:04.909582 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:04.909547    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:05.715607 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:05.715554    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:05.819841 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:05.819809    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-c78rn"]
Jul 24 17:49:05.821329 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:05.821301    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:05.822255 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:05.822238    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-c78rn"
Jul 24 17:49:05.824950 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:05.824930    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-t5zg4\""
Jul 24 17:49:05.840951 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:05.840929    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-c78rn"]
Jul 24 17:49:05.931511 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:05.931477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6chk\" (UniqueName: \"kubernetes.io/projected/375aec64-0cfe-4df5-a935-abe0f42db406-kube-api-access-w6chk\") pod \"authorino-operator-68d88948d8-c78rn\" (UID: \"375aec64-0cfe-4df5-a935-abe0f42db406\") " pod="kuadrant-system/authorino-operator-68d88948d8-c78rn"
Jul 24 17:49:06.032150 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:06.032109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6chk\" (UniqueName: \"kubernetes.io/projected/375aec64-0cfe-4df5-a935-abe0f42db406-kube-api-access-w6chk\") pod \"authorino-operator-68d88948d8-c78rn\" (UID: \"375aec64-0cfe-4df5-a935-abe0f42db406\") " pod="kuadrant-system/authorino-operator-68d88948d8-c78rn"
Jul 24 17:49:06.050653 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:06.050619    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6chk\" (UniqueName: \"kubernetes.io/projected/375aec64-0cfe-4df5-a935-abe0f42db406-kube-api-access-w6chk\") pod \"authorino-operator-68d88948d8-c78rn\" (UID: \"375aec64-0cfe-4df5-a935-abe0f42db406\") " pod="kuadrant-system/authorino-operator-68d88948d8-c78rn"
Jul 24 17:49:06.125619 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:06.125544    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:06.132445 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:06.132427    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-c78rn"
Jul 24 17:49:06.256497 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:06.256463    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:06.268225 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:06.268203    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-c78rn"]
Jul 24 17:49:06.271104 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:49:06.271080    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375aec64_0cfe_4df5_a935_abe0f42db406.slice/crio-c280f4e4a36b30806b0d3663e4418b6a8d135a0745a60c9f6eb187823937e7ae WatchSource:0}: Error finding container c280f4e4a36b30806b0d3663e4418b6a8d135a0745a60c9f6eb187823937e7ae: Status 404 returned error can't find the container with id c280f4e4a36b30806b0d3663e4418b6a8d135a0745a60c9f6eb187823937e7ae
Jul 24 17:49:06.528768 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:06.528671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-c78rn" event={"ID":"375aec64-0cfe-4df5-a935-abe0f42db406","Type":"ContainerStarted","Data":"c280f4e4a36b30806b0d3663e4418b6a8d135a0745a60c9f6eb187823937e7ae"}
Jul 24 17:49:06.530201 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:06.530171    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5tlqg" event={"ID":"4d0cf1b5-e21d-4eca-bacc-5fc111b0549a","Type":"ContainerStarted","Data":"e400ee20292a0763ffd6762500abcb29d8829476d55435f60be5b6e93f6f1821"}
Jul 24 17:49:06.530335 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:06.530315    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5tlqg"
Jul 24 17:49:06.549145 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:06.549101    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5tlqg" podStartSLOduration=1.95904627 podStartE2EDuration="3.549086852s" totalImagesPullingTime="1.590040582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:49:06.548211411 +0000 UTC m=+369.071569890" watchObservedRunningTime="2026-07-24 17:49:06.549086852 +0000 UTC m=+369.072445331"
Jul 24 17:49:06.854738 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:06.854705    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:07.122113 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:07.122040    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:07.229022 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:07.228992    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:07.324233 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:07.324204    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:07.961324 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:07.961293    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:08.061827 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:08.061799    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:08.541632 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:08.541598    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-c78rn" event={"ID":"375aec64-0cfe-4df5-a935-abe0f42db406","Type":"ContainerStarted","Data":"4bd0e460c48c59e0169cd8541747064f97840ba98737f56423f3e8e0af855328"}
Jul 24 17:49:08.541851 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:08.541718    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-c78rn"
Jul 24 17:49:08.559525 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:08.559482    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-c78rn" podStartSLOduration=1.8592001919999999 podStartE2EDuration="3.559470047s" totalImagesPullingTime="1.700269855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:49:08.558659942 +0000 UTC m=+371.082018431" watchObservedRunningTime="2026-07-24 17:49:08.559470047 +0000 UTC m=+371.082828535"
Jul 24 17:49:10.150898 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:10.150868    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"]
Jul 24 17:49:10.153072 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:10.153057    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"
Jul 24 17:49:10.157404 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:10.157385    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4znsw\""
Jul 24 17:49:10.174937 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:10.174913    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"]
Jul 24 17:49:10.269296 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:10.269257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nrk6\" (UniqueName: \"kubernetes.io/projected/9c3cf7f1-ea65-4476-b3c2-92ec843c48d2-kube-api-access-5nrk6\") pod \"kuadrant-operator-controller-manager-58fc56757c-qkhl5\" (UID: \"9c3cf7f1-ea65-4476-b3c2-92ec843c48d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"
Jul 24 17:49:10.269470 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:10.269332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c3cf7f1-ea65-4476-b3c2-92ec843c48d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-qkhl5\" (UID: \"9c3cf7f1-ea65-4476-b3c2-92ec843c48d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"
Jul 24 17:49:10.370317 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:10.370285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c3cf7f1-ea65-4476-b3c2-92ec843c48d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-qkhl5\" (UID: \"9c3cf7f1-ea65-4476-b3c2-92ec843c48d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"
Jul 24 17:49:10.370477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:10.370334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nrk6\" (UniqueName: \"kubernetes.io/projected/9c3cf7f1-ea65-4476-b3c2-92ec843c48d2-kube-api-access-5nrk6\") pod \"kuadrant-operator-controller-manager-58fc56757c-qkhl5\" (UID: \"9c3cf7f1-ea65-4476-b3c2-92ec843c48d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"
Jul 24 17:49:10.370651 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:10.370633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c3cf7f1-ea65-4476-b3c2-92ec843c48d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-qkhl5\" (UID: \"9c3cf7f1-ea65-4476-b3c2-92ec843c48d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"
Jul 24 17:49:10.379171 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:10.379145    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nrk6\" (UniqueName: \"kubernetes.io/projected/9c3cf7f1-ea65-4476-b3c2-92ec843c48d2-kube-api-access-5nrk6\") pod \"kuadrant-operator-controller-manager-58fc56757c-qkhl5\" (UID: \"9c3cf7f1-ea65-4476-b3c2-92ec843c48d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"
Jul 24 17:49:10.462807 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:10.462703    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"
Jul 24 17:49:10.597114 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:10.597079    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"]
Jul 24 17:49:10.599298 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:49:10.599268    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3cf7f1_ea65_4476_b3c2_92ec843c48d2.slice/crio-6a3e85345cc9430dd0b26032b8c97cf4dd3ff4cc07e6a4307b7692f0af8bef0c WatchSource:0}: Error finding container 6a3e85345cc9430dd0b26032b8c97cf4dd3ff4cc07e6a4307b7692f0af8bef0c: Status 404 returned error can't find the container with id 6a3e85345cc9430dd0b26032b8c97cf4dd3ff4cc07e6a4307b7692f0af8bef0c
Jul 24 17:49:11.557202 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:11.557156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5" event={"ID":"9c3cf7f1-ea65-4476-b3c2-92ec843c48d2","Type":"ContainerStarted","Data":"6a3e85345cc9430dd0b26032b8c97cf4dd3ff4cc07e6a4307b7692f0af8bef0c"}
Jul 24 17:49:12.279679 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:12.279647    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:12.377438 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:12.377402    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:14.284426 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:14.284395    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:14.345443 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:14.345408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:15.524326 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:15.524289    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt"
Jul 24 17:49:16.582170 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:16.582131    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5" event={"ID":"9c3cf7f1-ea65-4476-b3c2-92ec843c48d2","Type":"ContainerStarted","Data":"5f988a12b113b293b91bd7f568adc666d5a4f6905718a832c033c9d5101eec74"}
Jul 24 17:49:16.582565 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:16.582220    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"
Jul 24 17:49:16.606992 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:16.606946    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5" podStartSLOduration=0.989183226 podStartE2EDuration="6.606928631s" totalImagesPullingTime="5.617745405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:49:16.604155362 +0000 UTC m=+379.127513832" watchObservedRunningTime="2026-07-24 17:49:16.606928631 +0000 UTC m=+379.130287109"
Jul 24 17:49:17.538708 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:17.538668    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5tlqg"
Jul 24 17:49:18.903858 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:18.903828    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:19.129522 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:19.129494    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:19.548359 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:19.548332    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-c78rn"
Jul 24 17:49:20.188951 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:20.188917    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:20.292253 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:20.292209    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:24.837329 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:24.837297    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:24.937382 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:24.937352    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:25.038747 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:25.038719    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:27.589920 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:27.589888    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"
Jul 24 17:49:29.167980 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.167941    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw"]
Jul 24 17:49:29.174471 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.174442    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw"
Jul 24 17:49:29.187925 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.187894    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw"]
Jul 24 17:49:29.331025 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.330983    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f9d113d-5972-425c-b2b5-2116b1b26845-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-f56fw\" (UID: \"3f9d113d-5972-425c-b2b5-2116b1b26845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw"
Jul 24 17:49:29.331214 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.331058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw8sg\" (UniqueName: \"kubernetes.io/projected/3f9d113d-5972-425c-b2b5-2116b1b26845-kube-api-access-nw8sg\") pod \"kuadrant-operator-controller-manager-6bd956ccf-f56fw\" (UID: \"3f9d113d-5972-425c-b2b5-2116b1b26845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw"
Jul 24 17:49:29.431695 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.431616    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f9d113d-5972-425c-b2b5-2116b1b26845-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-f56fw\" (UID: \"3f9d113d-5972-425c-b2b5-2116b1b26845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw"
Jul 24 17:49:29.431695 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.431680    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw8sg\" (UniqueName: \"kubernetes.io/projected/3f9d113d-5972-425c-b2b5-2116b1b26845-kube-api-access-nw8sg\") pod \"kuadrant-operator-controller-manager-6bd956ccf-f56fw\" (UID: \"3f9d113d-5972-425c-b2b5-2116b1b26845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw"
Jul 24 17:49:29.432050 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.432030    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f9d113d-5972-425c-b2b5-2116b1b26845-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-f56fw\" (UID: \"3f9d113d-5972-425c-b2b5-2116b1b26845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw"
Jul 24 17:49:29.441177 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.441153    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw8sg\" (UniqueName: \"kubernetes.io/projected/3f9d113d-5972-425c-b2b5-2116b1b26845-kube-api-access-nw8sg\") pod \"kuadrant-operator-controller-manager-6bd956ccf-f56fw\" (UID: \"3f9d113d-5972-425c-b2b5-2116b1b26845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw"
Jul 24 17:49:29.477888 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.477855    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"]
Jul 24 17:49:29.478116 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.478079    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5" podUID="9c3cf7f1-ea65-4476-b3c2-92ec843c48d2" containerName="manager" containerID="cri-o://5f988a12b113b293b91bd7f568adc666d5a4f6905718a832c033c9d5101eec74" gracePeriod=2
Jul 24 17:49:29.484388 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.484365    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw"
Jul 24 17:49:29.489497 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.489475    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"]
Jul 24 17:49:29.489651 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:49:29.489499    2578 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"9c3cf7f1-ea65-4476-b3c2-92ec843c48d2\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-58fc56757c-qkhl5\": pods \"kuadrant-operator-controller-manager-58fc56757c-qkhl5\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"
Jul 24 17:49:29.507608 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.507532    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw"]
Jul 24 17:49:29.514150 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.510587    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw"]
Jul 24 17:49:29.514150 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.511149    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c3cf7f1-ea65-4476-b3c2-92ec843c48d2" containerName="manager"
Jul 24 17:49:29.514150 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.511168    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c3cf7f1-ea65-4476-b3c2-92ec843c48d2" containerName="manager"
Jul 24 17:49:29.514150 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.511264    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c3cf7f1-ea65-4476-b3c2-92ec843c48d2" containerName="manager"
Jul 24 17:49:29.516158 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.516129    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw"]
Jul 24 17:49:29.516281 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.516241    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw"
Jul 24 17:49:29.518756 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:49:29.518725    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-qkhl5\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" podUID="9c3cf7f1-ea65-4476-b3c2-92ec843c48d2" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"
Jul 24 17:49:29.524838 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.524815    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw"]
Jul 24 17:49:29.542077 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.542050    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9"]
Jul 24 17:49:29.545686 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.545665    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9"
Jul 24 17:49:29.562291 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.562264    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt"]
Jul 24 17:49:29.562478 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.562454    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt" podUID="c36f646e-eb2b-4f6e-9003-1010bd8418ca" containerName="manager" containerID="cri-o://0460454553a68b60a1fe4d63222899796f1b717b261f925acafeef4182785322" gracePeriod=2
Jul 24 17:49:29.564634 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.564554    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9"]
Jul 24 17:49:29.566218 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.566179    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt"]
Jul 24 17:49:29.594119 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:49:29.594076    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-qkhl5\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" podUID="9c3cf7f1-ea65-4476-b3c2-92ec843c48d2" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"
Jul 24 17:49:29.596014 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.595989    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mq8wk"]
Jul 24 17:49:29.596311 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:49:29.596279    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-zvbpt\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" podUID="c36f646e-eb2b-4f6e-9003-1010bd8418ca" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt"
Jul 24 17:49:29.596439 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.596421    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c36f646e-eb2b-4f6e-9003-1010bd8418ca" containerName="manager"
Jul 24 17:49:29.596486 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.596445    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c36f646e-eb2b-4f6e-9003-1010bd8418ca" containerName="manager"
Jul 24 17:49:29.596553 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.596541    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c36f646e-eb2b-4f6e-9003-1010bd8418ca" containerName="manager"
Jul 24 17:49:29.598344 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:49:29.598322    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-qkhl5\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" podUID="9c3cf7f1-ea65-4476-b3c2-92ec843c48d2" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"
Jul 24 17:49:29.599821 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.599804    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mq8wk"
Jul 24 17:49:29.602466 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:49:29.602445    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-qkhl5\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" podUID="9c3cf7f1-ea65-4476-b3c2-92ec843c48d2" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"
Jul 24 17:49:29.605236 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:49:29.605209    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-zvbpt\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" podUID="c36f646e-eb2b-4f6e-9003-1010bd8418ca" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt"
Jul 24 17:49:29.607766 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.607744    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mq8wk"]
Jul 24 17:49:29.634839 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.634810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8vjq\" (UniqueName: \"kubernetes.io/projected/561c4b23-4286-4f3e-948d-a6aa35d020f2-kube-api-access-m8vjq\") pod \"kuadrant-operator-controller-manager-6bd956ccf-xfbk9\" (UID: \"561c4b23-4286-4f3e-948d-a6aa35d020f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9"
Jul 24 17:49:29.634992 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.634860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/561c4b23-4286-4f3e-948d-a6aa35d020f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-xfbk9\" (UID: \"561c4b23-4286-4f3e-948d-a6aa35d020f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9"
Jul 24 17:49:29.634992 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.634917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a358961-26f3-4af0-b575-495f6900be9d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-tr4tw\" (UID: \"7a358961-26f3-4af0-b575-495f6900be9d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw"
Jul 24 17:49:29.634992 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.634990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8bpn\" (UniqueName: \"kubernetes.io/projected/7a358961-26f3-4af0-b575-495f6900be9d-kube-api-access-w8bpn\") pod \"kuadrant-operator-controller-manager-58fc56757c-tr4tw\" (UID: \"7a358961-26f3-4af0-b575-495f6900be9d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw"
Jul 24 17:49:29.650728 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.650705    2578 generic.go:356] "Generic (PLEG): container finished" podID="9c3cf7f1-ea65-4476-b3c2-92ec843c48d2" containerID="5f988a12b113b293b91bd7f568adc666d5a4f6905718a832c033c9d5101eec74" exitCode=0
Jul 24 17:49:29.738445 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.737311    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nks2\" (UniqueName: \"kubernetes.io/projected/f99f537d-8bc5-47e1-9f76-8af38e631178-kube-api-access-4nks2\") pod \"limitador-operator-controller-manager-6c549b6d85-mq8wk\" (UID: \"f99f537d-8bc5-47e1-9f76-8af38e631178\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mq8wk"
Jul 24 17:49:29.738445 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.737391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8bpn\" (UniqueName: \"kubernetes.io/projected/7a358961-26f3-4af0-b575-495f6900be9d-kube-api-access-w8bpn\") pod \"kuadrant-operator-controller-manager-58fc56757c-tr4tw\" (UID: \"7a358961-26f3-4af0-b575-495f6900be9d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw"
Jul 24 17:49:29.738445 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.737489    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8vjq\" (UniqueName: \"kubernetes.io/projected/561c4b23-4286-4f3e-948d-a6aa35d020f2-kube-api-access-m8vjq\") pod \"kuadrant-operator-controller-manager-6bd956ccf-xfbk9\" (UID: \"561c4b23-4286-4f3e-948d-a6aa35d020f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9"
Jul 24 17:49:29.738445 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.737521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/561c4b23-4286-4f3e-948d-a6aa35d020f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-xfbk9\" (UID: \"561c4b23-4286-4f3e-948d-a6aa35d020f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9"
Jul 24 17:49:29.738445 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.737552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a358961-26f3-4af0-b575-495f6900be9d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-tr4tw\" (UID: \"7a358961-26f3-4af0-b575-495f6900be9d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw"
Jul 24 17:49:29.738445 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.738109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a358961-26f3-4af0-b575-495f6900be9d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-tr4tw\" (UID: \"7a358961-26f3-4af0-b575-495f6900be9d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw"
Jul 24 17:49:29.738445 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.738397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/561c4b23-4286-4f3e-948d-a6aa35d020f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-xfbk9\" (UID: \"561c4b23-4286-4f3e-948d-a6aa35d020f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9"
Jul 24 17:49:29.752008 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.751951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8vjq\" (UniqueName: \"kubernetes.io/projected/561c4b23-4286-4f3e-948d-a6aa35d020f2-kube-api-access-m8vjq\") pod \"kuadrant-operator-controller-manager-6bd956ccf-xfbk9\" (UID: \"561c4b23-4286-4f3e-948d-a6aa35d020f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9"
Jul 24 17:49:29.753252 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.753207    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8bpn\" (UniqueName: \"kubernetes.io/projected/7a358961-26f3-4af0-b575-495f6900be9d-kube-api-access-w8bpn\") pod \"kuadrant-operator-controller-manager-58fc56757c-tr4tw\" (UID: \"7a358961-26f3-4af0-b575-495f6900be9d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw"
Jul 24 17:49:29.808305 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.808280    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"
Jul 24 17:49:29.810684 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:49:29.810661    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-qkhl5\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" podUID="9c3cf7f1-ea65-4476-b3c2-92ec843c48d2" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"
Jul 24 17:49:29.811087 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.811070    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt"
Jul 24 17:49:29.812769 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:49:29.812744    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-zvbpt\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" podUID="c36f646e-eb2b-4f6e-9003-1010bd8418ca" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt"
Jul 24 17:49:29.814857 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:49:29.814834    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-zvbpt\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" podUID="c36f646e-eb2b-4f6e-9003-1010bd8418ca" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt"
Jul 24 17:49:29.816920 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:49:29.816904    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-qkhl5\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" podUID="9c3cf7f1-ea65-4476-b3c2-92ec843c48d2" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"
Jul 24 17:49:29.838383 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.838344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nks2\" (UniqueName: \"kubernetes.io/projected/f99f537d-8bc5-47e1-9f76-8af38e631178-kube-api-access-4nks2\") pod \"limitador-operator-controller-manager-6c549b6d85-mq8wk\" (UID: \"f99f537d-8bc5-47e1-9f76-8af38e631178\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mq8wk"
Jul 24 17:49:29.848912 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.848891    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nks2\" (UniqueName: \"kubernetes.io/projected/f99f537d-8bc5-47e1-9f76-8af38e631178-kube-api-access-4nks2\") pod \"limitador-operator-controller-manager-6c549b6d85-mq8wk\" (UID: \"f99f537d-8bc5-47e1-9f76-8af38e631178\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mq8wk"
Jul 24 17:49:29.855830 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.855806    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw"
Jul 24 17:49:29.862580 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.862556    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9"
Jul 24 17:49:29.939551 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.939430    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mq8wk"
Jul 24 17:49:29.939722 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.939589    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nrk6\" (UniqueName: \"kubernetes.io/projected/9c3cf7f1-ea65-4476-b3c2-92ec843c48d2-kube-api-access-5nrk6\") pod \"9c3cf7f1-ea65-4476-b3c2-92ec843c48d2\" (UID: \"9c3cf7f1-ea65-4476-b3c2-92ec843c48d2\") "
Jul 24 17:49:29.939722 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.939679    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrqvb\" (UniqueName: \"kubernetes.io/projected/c36f646e-eb2b-4f6e-9003-1010bd8418ca-kube-api-access-wrqvb\") pod \"c36f646e-eb2b-4f6e-9003-1010bd8418ca\" (UID: \"c36f646e-eb2b-4f6e-9003-1010bd8418ca\") "
Jul 24 17:49:29.939875 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.939764    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c3cf7f1-ea65-4476-b3c2-92ec843c48d2-extensions-socket-volume\") pod \"9c3cf7f1-ea65-4476-b3c2-92ec843c48d2\" (UID: \"9c3cf7f1-ea65-4476-b3c2-92ec843c48d2\") "
Jul 24 17:49:29.941087 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.940527    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3cf7f1-ea65-4476-b3c2-92ec843c48d2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9c3cf7f1-ea65-4476-b3c2-92ec843c48d2" (UID: "9c3cf7f1-ea65-4476-b3c2-92ec843c48d2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:49:29.942332 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.942309    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c36f646e-eb2b-4f6e-9003-1010bd8418ca-kube-api-access-wrqvb" (OuterVolumeSpecName: "kube-api-access-wrqvb") pod "c36f646e-eb2b-4f6e-9003-1010bd8418ca" (UID: "c36f646e-eb2b-4f6e-9003-1010bd8418ca"). InnerVolumeSpecName "kube-api-access-wrqvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:49:29.942544 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.942512    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3cf7f1-ea65-4476-b3c2-92ec843c48d2-kube-api-access-5nrk6" (OuterVolumeSpecName: "kube-api-access-5nrk6") pod "9c3cf7f1-ea65-4476-b3c2-92ec843c48d2" (UID: "9c3cf7f1-ea65-4476-b3c2-92ec843c48d2"). InnerVolumeSpecName "kube-api-access-5nrk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:49:29.968810 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.968596    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c3cf7f1-ea65-4476-b3c2-92ec843c48d2" path="/var/lib/kubelet/pods/9c3cf7f1-ea65-4476-b3c2-92ec843c48d2/volumes"
Jul 24 17:49:29.968981 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:29.968962    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c36f646e-eb2b-4f6e-9003-1010bd8418ca" path="/var/lib/kubelet/pods/c36f646e-eb2b-4f6e-9003-1010bd8418ca/volumes"
Jul 24 17:49:30.021019 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.020981    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw"]
Jul 24 17:49:30.024820 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:49:30.024771    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a358961_26f3_4af0_b575_495f6900be9d.slice/crio-075737cd15b9b93c1d479e6bc7091e79e4d40f67dd4287c235bddce08fff68e3 WatchSource:0}: Error finding container 075737cd15b9b93c1d479e6bc7091e79e4d40f67dd4287c235bddce08fff68e3: Status 404 returned error can't find the container with id 075737cd15b9b93c1d479e6bc7091e79e4d40f67dd4287c235bddce08fff68e3
Jul 24 17:49:30.040677 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.040625    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9"]
Jul 24 17:49:30.041075 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.041054    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nrk6\" (UniqueName: \"kubernetes.io/projected/9c3cf7f1-ea65-4476-b3c2-92ec843c48d2-kube-api-access-5nrk6\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:49:30.041161 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.041081    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrqvb\" (UniqueName: \"kubernetes.io/projected/c36f646e-eb2b-4f6e-9003-1010bd8418ca-kube-api-access-wrqvb\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:49:30.041161 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.041097    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c3cf7f1-ea65-4476-b3c2-92ec843c48d2-extensions-socket-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:49:30.043495 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:49:30.043463    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod561c4b23_4286_4f3e_948d_a6aa35d020f2.slice/crio-e1d8c60dd4b1fe424101b165601e898f081242d1d0ad51492c6c075819cb67fd WatchSource:0}: Error finding container e1d8c60dd4b1fe424101b165601e898f081242d1d0ad51492c6c075819cb67fd: Status 404 returned error can't find the container with id e1d8c60dd4b1fe424101b165601e898f081242d1d0ad51492c6c075819cb67fd
Jul 24 17:49:30.104068 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.104009    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mq8wk"]
Jul 24 17:49:30.106143 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:49:30.106117    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99f537d_8bc5_47e1_9f76_8af38e631178.slice/crio-73c0fe85bcd76aee6383dabdd0e776dc004432db2a25d4bc11e3c4fe6c543593 WatchSource:0}: Error finding container 73c0fe85bcd76aee6383dabdd0e776dc004432db2a25d4bc11e3c4fe6c543593: Status 404 returned error can't find the container with id 73c0fe85bcd76aee6383dabdd0e776dc004432db2a25d4bc11e3c4fe6c543593
Jul 24 17:49:30.500063 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:49:30.500021    2578 log.go:32] "RunPodSandbox from runtime service failed" err=<
Jul 24 17:49:30.500063 ip-10-0-137-93 kubenswrapper[2578]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6bd956ccf-f56fw_kuadrant-system_3f9d113d-5972-425c-b2b5-2116b1b26845_0(3f241716c8d51dcb502b9e5f3621199c7fa38ddffeb1e14a7308a2e92aa056fd): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6bd956ccf-f56fw to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3f241716c8d51dcb502b9e5f3621199c7fa38ddffeb1e14a7308a2e92aa056fd" Netns:"/var/run/netns/f77b84df-433b-4f48-8f56-5e03e90eafa2" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-6bd956ccf-f56fw;K8S_POD_INFRA_CONTAINER_ID=3f241716c8d51dcb502b9e5f3621199c7fa38ddffeb1e14a7308a2e92aa056fd;K8S_POD_UID=3f9d113d-5972-425c-b2b5-2116b1b26845" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw] networking: [kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw/3f9d113d-5972-425c-b2b5-2116b1b26845:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw 3f241716c8d51dcb502b9e5f3621199c7fa38ddffeb1e14a7308a2e92aa056fd network default NAD default NAD key default] failed to get pod kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw: timed out waiting for pod after 1s: pods "kuadrant-operator-controller-manager-6bd956ccf-f56fw" not found
Jul 24 17:49:30.500063 ip-10-0-137-93 kubenswrapper[2578]:         '
Jul 24 17:49:30.500063 ip-10-0-137-93 kubenswrapper[2578]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 24 17:49:30.500063 ip-10-0-137-93 kubenswrapper[2578]:  >
Jul 24 17:49:30.500635 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:49:30.500101    2578 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Jul 24 17:49:30.500635 ip-10-0-137-93 kubenswrapper[2578]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6bd956ccf-f56fw_kuadrant-system_3f9d113d-5972-425c-b2b5-2116b1b26845_0(3f241716c8d51dcb502b9e5f3621199c7fa38ddffeb1e14a7308a2e92aa056fd): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6bd956ccf-f56fw to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3f241716c8d51dcb502b9e5f3621199c7fa38ddffeb1e14a7308a2e92aa056fd" Netns:"/var/run/netns/f77b84df-433b-4f48-8f56-5e03e90eafa2" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-6bd956ccf-f56fw;K8S_POD_INFRA_CONTAINER_ID=3f241716c8d51dcb502b9e5f3621199c7fa38ddffeb1e14a7308a2e92aa056fd;K8S_POD_UID=3f9d113d-5972-425c-b2b5-2116b1b26845" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw] networking: [kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw/3f9d113d-5972-425c-b2b5-2116b1b26845:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw 3f241716c8d51dcb502b9e5f3621199c7fa38ddffeb1e14a7308a2e92aa056fd network default NAD default NAD key default] failed to get pod kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw: timed out waiting for pod after 1s: pods "kuadrant-operator-controller-manager-6bd956ccf-f56fw" not found
Jul 24 17:49:30.500635 ip-10-0-137-93 kubenswrapper[2578]:         '
Jul 24 17:49:30.500635 ip-10-0-137-93 kubenswrapper[2578]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 24 17:49:30.500635 ip-10-0-137-93 kubenswrapper[2578]:  > pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw"
Jul 24 17:49:30.656548 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.656520    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qkhl5"
Jul 24 17:49:30.656548 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.656549    2578 scope.go:117] "RemoveContainer" containerID="5f988a12b113b293b91bd7f568adc666d5a4f6905718a832c033c9d5101eec74"
Jul 24 17:49:30.658520 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.658379    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mq8wk" event={"ID":"f99f537d-8bc5-47e1-9f76-8af38e631178","Type":"ContainerStarted","Data":"a1ddb307f2fdeee84ca3b360445d46d5a3b4998b5da206989db352bbbc3c8c9d"}
Jul 24 17:49:30.658520 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.658437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mq8wk" event={"ID":"f99f537d-8bc5-47e1-9f76-8af38e631178","Type":"ContainerStarted","Data":"73c0fe85bcd76aee6383dabdd0e776dc004432db2a25d4bc11e3c4fe6c543593"}
Jul 24 17:49:30.658520 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.658492    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mq8wk"
Jul 24 17:49:30.660126 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.660103    2578 generic.go:356] "Generic (PLEG): container finished" podID="c36f646e-eb2b-4f6e-9003-1010bd8418ca" containerID="0460454553a68b60a1fe4d63222899796f1b717b261f925acafeef4182785322" exitCode=0
Jul 24 17:49:30.660230 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.660152    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zvbpt"
Jul 24 17:49:30.661891 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.661864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw" event={"ID":"7a358961-26f3-4af0-b575-495f6900be9d","Type":"ContainerStarted","Data":"2858574e1f7fc96e073c99aaeceef2abe809a6c13254a27ce5967bcd106d818d"}
Jul 24 17:49:30.661975 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.661916    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw" event={"ID":"7a358961-26f3-4af0-b575-495f6900be9d","Type":"ContainerStarted","Data":"075737cd15b9b93c1d479e6bc7091e79e4d40f67dd4287c235bddce08fff68e3"}
Jul 24 17:49:30.662392 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.662369    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw"
Jul 24 17:49:30.664036 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.664020    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw"
Jul 24 17:49:30.664146 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.664067    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9" event={"ID":"561c4b23-4286-4f3e-948d-a6aa35d020f2","Type":"ContainerStarted","Data":"d88d3c3abf1d126fddccc481d97750465061e2aca2284c8f2e6541d9b6a8b29e"}
Jul 24 17:49:30.664146 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.664111    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9" event={"ID":"561c4b23-4286-4f3e-948d-a6aa35d020f2","Type":"ContainerStarted","Data":"e1d8c60dd4b1fe424101b165601e898f081242d1d0ad51492c6c075819cb67fd"}
Jul 24 17:49:30.666912 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.666895    2578 scope.go:117] "RemoveContainer" containerID="0460454553a68b60a1fe4d63222899796f1b717b261f925acafeef4182785322"
Jul 24 17:49:30.672420 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.672377    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw"
Jul 24 17:49:30.674341 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.674321    2578 scope.go:117] "RemoveContainer" containerID="0460454553a68b60a1fe4d63222899796f1b717b261f925acafeef4182785322"
Jul 24 17:49:30.674589 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:49:30.674571    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0460454553a68b60a1fe4d63222899796f1b717b261f925acafeef4182785322\": container with ID starting with 0460454553a68b60a1fe4d63222899796f1b717b261f925acafeef4182785322 not found: ID does not exist" containerID="0460454553a68b60a1fe4d63222899796f1b717b261f925acafeef4182785322"
Jul 24 17:49:30.674647 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.674597    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0460454553a68b60a1fe4d63222899796f1b717b261f925acafeef4182785322"} err="failed to get container status \"0460454553a68b60a1fe4d63222899796f1b717b261f925acafeef4182785322\": rpc error: code = NotFound desc = could not find container \"0460454553a68b60a1fe4d63222899796f1b717b261f925acafeef4182785322\": container with ID starting with 0460454553a68b60a1fe4d63222899796f1b717b261f925acafeef4182785322 not found: ID does not exist"
Jul 24 17:49:30.689813 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.689751    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mq8wk" podStartSLOduration=1.68973829 podStartE2EDuration="1.68973829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:49:30.688861462 +0000 UTC m=+393.212219941" watchObservedRunningTime="2026-07-24 17:49:30.68973829 +0000 UTC m=+393.213096745"
Jul 24 17:49:30.740309 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.740253    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw" podStartSLOduration=1.740237815 podStartE2EDuration="1.740237815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:49:30.737746182 +0000 UTC m=+393.261104660" watchObservedRunningTime="2026-07-24 17:49:30.740237815 +0000 UTC m=+393.263596292"
Jul 24 17:49:30.775138 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.775090    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9" podStartSLOduration=1.775074933 podStartE2EDuration="1.775074933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:49:30.773747434 +0000 UTC m=+393.297105913" watchObservedRunningTime="2026-07-24 17:49:30.775074933 +0000 UTC m=+393.298433411"
Jul 24 17:49:30.777096 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:49:30.777070    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-f56fw\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" podUID="3f9d113d-5972-425c-b2b5-2116b1b26845" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw"
Jul 24 17:49:30.848426 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.848393    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f9d113d-5972-425c-b2b5-2116b1b26845-extensions-socket-volume\") pod \"3f9d113d-5972-425c-b2b5-2116b1b26845\" (UID: \"3f9d113d-5972-425c-b2b5-2116b1b26845\") "
Jul 24 17:49:30.848610 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.848463    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw8sg\" (UniqueName: \"kubernetes.io/projected/3f9d113d-5972-425c-b2b5-2116b1b26845-kube-api-access-nw8sg\") pod \"3f9d113d-5972-425c-b2b5-2116b1b26845\" (UID: \"3f9d113d-5972-425c-b2b5-2116b1b26845\") "
Jul 24 17:49:30.848674 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.848634    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9d113d-5972-425c-b2b5-2116b1b26845-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3f9d113d-5972-425c-b2b5-2116b1b26845" (UID: "3f9d113d-5972-425c-b2b5-2116b1b26845"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:49:30.848733 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.848711    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f9d113d-5972-425c-b2b5-2116b1b26845-extensions-socket-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:49:30.850531 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.850508    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9d113d-5972-425c-b2b5-2116b1b26845-kube-api-access-nw8sg" (OuterVolumeSpecName: "kube-api-access-nw8sg") pod "3f9d113d-5972-425c-b2b5-2116b1b26845" (UID: "3f9d113d-5972-425c-b2b5-2116b1b26845"). InnerVolumeSpecName "kube-api-access-nw8sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:49:30.949598 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:30.949518    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw8sg\" (UniqueName: \"kubernetes.io/projected/3f9d113d-5972-425c-b2b5-2116b1b26845-kube-api-access-nw8sg\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:49:31.670221 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:31.670192    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw"
Jul 24 17:49:31.670767 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:31.670741    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9"
Jul 24 17:49:31.685650 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:49:31.685618    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-f56fw\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" podUID="3f9d113d-5972-425c-b2b5-2116b1b26845" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw"
Jul 24 17:49:31.688229 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:49:31.688204    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-f56fw\" is forbidden: User \"system:node:ip-10-0-137-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-93.ec2.internal' and this object" podUID="3f9d113d-5972-425c-b2b5-2116b1b26845" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-f56fw"
Jul 24 17:49:31.720185 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:31.720162    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:31.946007 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:31.945920    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:31.968169 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:31.968139    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f9d113d-5972-425c-b2b5-2116b1b26845" path="/var/lib/kubelet/pods/3f9d113d-5972-425c-b2b5-2116b1b26845/volumes"
Jul 24 17:49:34.332622 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:34.332587    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:34.432478 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:34.432453    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:35.768884 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:35.768813    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:39.247440 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:39.247404    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:39.345263 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:39.345234    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:39.415257 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:39.415231    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:41.672727 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:41.672691    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mq8wk"
Jul 24 17:49:42.677461 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:42.677430    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9"
Jul 24 17:49:42.677890 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:42.677511    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw"
Jul 24 17:49:42.774445 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:42.774403    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw"]
Jul 24 17:49:42.774674 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:42.774648    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw" podUID="7a358961-26f3-4af0-b575-495f6900be9d" containerName="manager" containerID="cri-o://2858574e1f7fc96e073c99aaeceef2abe809a6c13254a27ce5967bcd106d818d" gracePeriod=10
Jul 24 17:49:43.025554 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.025531    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw"
Jul 24 17:49:43.048254 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.048232    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8bpn\" (UniqueName: \"kubernetes.io/projected/7a358961-26f3-4af0-b575-495f6900be9d-kube-api-access-w8bpn\") pod \"7a358961-26f3-4af0-b575-495f6900be9d\" (UID: \"7a358961-26f3-4af0-b575-495f6900be9d\") "
Jul 24 17:49:43.048368 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.048272    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a358961-26f3-4af0-b575-495f6900be9d-extensions-socket-volume\") pod \"7a358961-26f3-4af0-b575-495f6900be9d\" (UID: \"7a358961-26f3-4af0-b575-495f6900be9d\") "
Jul 24 17:49:43.048636 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.048606    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a358961-26f3-4af0-b575-495f6900be9d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7a358961-26f3-4af0-b575-495f6900be9d" (UID: "7a358961-26f3-4af0-b575-495f6900be9d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:49:43.050316 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.050296    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a358961-26f3-4af0-b575-495f6900be9d-kube-api-access-w8bpn" (OuterVolumeSpecName: "kube-api-access-w8bpn") pod "7a358961-26f3-4af0-b575-495f6900be9d" (UID: "7a358961-26f3-4af0-b575-495f6900be9d"). InnerVolumeSpecName "kube-api-access-w8bpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:49:43.121904 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.121873    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr"]
Jul 24 17:49:43.122266 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.122252    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a358961-26f3-4af0-b575-495f6900be9d" containerName="manager"
Jul 24 17:49:43.122313 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.122268    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a358961-26f3-4af0-b575-495f6900be9d" containerName="manager"
Jul 24 17:49:43.122360 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.122349    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a358961-26f3-4af0-b575-495f6900be9d" containerName="manager"
Jul 24 17:49:43.124232 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.124213    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr"
Jul 24 17:49:43.149066 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.149039    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2n4c\" (UniqueName: \"kubernetes.io/projected/6b05503a-70dd-4d52-84a2-97214db91af3-kube-api-access-f2n4c\") pod \"kuadrant-operator-controller-manager-68cf589589-lwtxr\" (UID: \"6b05503a-70dd-4d52-84a2-97214db91af3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr"
Jul 24 17:49:43.149184 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.149093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b05503a-70dd-4d52-84a2-97214db91af3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-lwtxr\" (UID: \"6b05503a-70dd-4d52-84a2-97214db91af3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr"
Jul 24 17:49:43.149251 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.149218    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8bpn\" (UniqueName: \"kubernetes.io/projected/7a358961-26f3-4af0-b575-495f6900be9d-kube-api-access-w8bpn\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:49:43.149251 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.149236    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a358961-26f3-4af0-b575-495f6900be9d-extensions-socket-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:49:43.158434 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.158404    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr"]
Jul 24 17:49:43.250674 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.250600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2n4c\" (UniqueName: \"kubernetes.io/projected/6b05503a-70dd-4d52-84a2-97214db91af3-kube-api-access-f2n4c\") pod \"kuadrant-operator-controller-manager-68cf589589-lwtxr\" (UID: \"6b05503a-70dd-4d52-84a2-97214db91af3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr"
Jul 24 17:49:43.250674 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.250642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b05503a-70dd-4d52-84a2-97214db91af3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-lwtxr\" (UID: \"6b05503a-70dd-4d52-84a2-97214db91af3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr"
Jul 24 17:49:43.251087 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.251068    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b05503a-70dd-4d52-84a2-97214db91af3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-lwtxr\" (UID: \"6b05503a-70dd-4d52-84a2-97214db91af3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr"
Jul 24 17:49:43.259362 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.259336    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2n4c\" (UniqueName: \"kubernetes.io/projected/6b05503a-70dd-4d52-84a2-97214db91af3-kube-api-access-f2n4c\") pod \"kuadrant-operator-controller-manager-68cf589589-lwtxr\" (UID: \"6b05503a-70dd-4d52-84a2-97214db91af3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr"
Jul 24 17:49:43.434510 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.434478    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr"
Jul 24 17:49:43.573376 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.573347    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr"]
Jul 24 17:49:43.575404 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:49:43.575376    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b05503a_70dd_4d52_84a2_97214db91af3.slice/crio-cfec97cb0d5cd197b48e2f2ca3121db024a7f3640290515ec44929094546b8c8 WatchSource:0}: Error finding container cfec97cb0d5cd197b48e2f2ca3121db024a7f3640290515ec44929094546b8c8: Status 404 returned error can't find the container with id cfec97cb0d5cd197b48e2f2ca3121db024a7f3640290515ec44929094546b8c8
Jul 24 17:49:43.728435 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.728396    2578 generic.go:356] "Generic (PLEG): container finished" podID="7a358961-26f3-4af0-b575-495f6900be9d" containerID="2858574e1f7fc96e073c99aaeceef2abe809a6c13254a27ce5967bcd106d818d" exitCode=0
Jul 24 17:49:43.728891 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.728444    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw"
Jul 24 17:49:43.728891 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.728486    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw" event={"ID":"7a358961-26f3-4af0-b575-495f6900be9d","Type":"ContainerDied","Data":"2858574e1f7fc96e073c99aaeceef2abe809a6c13254a27ce5967bcd106d818d"}
Jul 24 17:49:43.728891 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.728543    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw" event={"ID":"7a358961-26f3-4af0-b575-495f6900be9d","Type":"ContainerDied","Data":"075737cd15b9b93c1d479e6bc7091e79e4d40f67dd4287c235bddce08fff68e3"}
Jul 24 17:49:43.728891 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.728579    2578 scope.go:117] "RemoveContainer" containerID="2858574e1f7fc96e073c99aaeceef2abe809a6c13254a27ce5967bcd106d818d"
Jul 24 17:49:43.730407 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.730383    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr" event={"ID":"6b05503a-70dd-4d52-84a2-97214db91af3","Type":"ContainerStarted","Data":"d4b8f40cdab48d96e66ea112f27edd8db7a5afcafad41ad63050ff5955eebfac"}
Jul 24 17:49:43.730504 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.730433    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr" event={"ID":"6b05503a-70dd-4d52-84a2-97214db91af3","Type":"ContainerStarted","Data":"cfec97cb0d5cd197b48e2f2ca3121db024a7f3640290515ec44929094546b8c8"}
Jul 24 17:49:43.730504 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.730468    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr"
Jul 24 17:49:43.739850 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.739831    2578 scope.go:117] "RemoveContainer" containerID="2858574e1f7fc96e073c99aaeceef2abe809a6c13254a27ce5967bcd106d818d"
Jul 24 17:49:43.740275 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:49:43.740252    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2858574e1f7fc96e073c99aaeceef2abe809a6c13254a27ce5967bcd106d818d\": container with ID starting with 2858574e1f7fc96e073c99aaeceef2abe809a6c13254a27ce5967bcd106d818d not found: ID does not exist" containerID="2858574e1f7fc96e073c99aaeceef2abe809a6c13254a27ce5967bcd106d818d"
Jul 24 17:49:43.740396 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.740280    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2858574e1f7fc96e073c99aaeceef2abe809a6c13254a27ce5967bcd106d818d"} err="failed to get container status \"2858574e1f7fc96e073c99aaeceef2abe809a6c13254a27ce5967bcd106d818d\": rpc error: code = NotFound desc = could not find container \"2858574e1f7fc96e073c99aaeceef2abe809a6c13254a27ce5967bcd106d818d\": container with ID starting with 2858574e1f7fc96e073c99aaeceef2abe809a6c13254a27ce5967bcd106d818d not found: ID does not exist"
Jul 24 17:49:43.760813 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.760754    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr" podStartSLOduration=0.760742242 podStartE2EDuration="760.742242ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:49:43.759843764 +0000 UTC m=+406.283202245" watchObservedRunningTime="2026-07-24 17:49:43.760742242 +0000 UTC m=+406.284100719"
Jul 24 17:49:43.783018 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.782961    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw"]
Jul 24 17:49:43.786957 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.786935    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tr4tw"]
Jul 24 17:49:43.966727 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:43.966696    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a358961-26f3-4af0-b575-495f6900be9d" path="/var/lib/kubelet/pods/7a358961-26f3-4af0-b575-495f6900be9d/volumes"
Jul 24 17:49:44.451085 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:44.451055    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:44.555129 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:44.555098    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:44.652208 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:44.652174    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:54.615118 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:54.615081    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:54.716372 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:54.716335    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:54.737592 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:54.737569    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr"
Jul 24 17:49:54.806613 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:54.806577    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9"]
Jul 24 17:49:54.806868 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:54.806844    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9" podUID="561c4b23-4286-4f3e-948d-a6aa35d020f2" containerName="manager" containerID="cri-o://d88d3c3abf1d126fddccc481d97750465061e2aca2284c8f2e6541d9b6a8b29e" gracePeriod=10
Jul 24 17:49:54.809589 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:54.809560    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:55.052164 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:55.052138    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9"
Jul 24 17:49:55.153758 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:55.153719    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/561c4b23-4286-4f3e-948d-a6aa35d020f2-extensions-socket-volume\") pod \"561c4b23-4286-4f3e-948d-a6aa35d020f2\" (UID: \"561c4b23-4286-4f3e-948d-a6aa35d020f2\") "
Jul 24 17:49:55.153949 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:55.153768    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8vjq\" (UniqueName: \"kubernetes.io/projected/561c4b23-4286-4f3e-948d-a6aa35d020f2-kube-api-access-m8vjq\") pod \"561c4b23-4286-4f3e-948d-a6aa35d020f2\" (UID: \"561c4b23-4286-4f3e-948d-a6aa35d020f2\") "
Jul 24 17:49:55.154133 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:55.154110    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/561c4b23-4286-4f3e-948d-a6aa35d020f2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "561c4b23-4286-4f3e-948d-a6aa35d020f2" (UID: "561c4b23-4286-4f3e-948d-a6aa35d020f2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:49:55.155864 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:55.155833    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/561c4b23-4286-4f3e-948d-a6aa35d020f2-kube-api-access-m8vjq" (OuterVolumeSpecName: "kube-api-access-m8vjq") pod "561c4b23-4286-4f3e-948d-a6aa35d020f2" (UID: "561c4b23-4286-4f3e-948d-a6aa35d020f2"). InnerVolumeSpecName "kube-api-access-m8vjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:49:55.255510 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:55.255427    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/561c4b23-4286-4f3e-948d-a6aa35d020f2-extensions-socket-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:49:55.255510 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:55.255455    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8vjq\" (UniqueName: \"kubernetes.io/projected/561c4b23-4286-4f3e-948d-a6aa35d020f2-kube-api-access-m8vjq\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:49:55.357179 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:55.357152    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:55.439361 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:55.439332    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:55.789199 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:55.789168    2578 generic.go:356] "Generic (PLEG): container finished" podID="561c4b23-4286-4f3e-948d-a6aa35d020f2" containerID="d88d3c3abf1d126fddccc481d97750465061e2aca2284c8f2e6541d9b6a8b29e" exitCode=0
Jul 24 17:49:55.789574 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:55.789215    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9"
Jul 24 17:49:55.789574 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:55.789240    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9" event={"ID":"561c4b23-4286-4f3e-948d-a6aa35d020f2","Type":"ContainerDied","Data":"d88d3c3abf1d126fddccc481d97750465061e2aca2284c8f2e6541d9b6a8b29e"}
Jul 24 17:49:55.789574 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:55.789280    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9" event={"ID":"561c4b23-4286-4f3e-948d-a6aa35d020f2","Type":"ContainerDied","Data":"e1d8c60dd4b1fe424101b165601e898f081242d1d0ad51492c6c075819cb67fd"}
Jul 24 17:49:55.789574 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:55.789301    2578 scope.go:117] "RemoveContainer" containerID="d88d3c3abf1d126fddccc481d97750465061e2aca2284c8f2e6541d9b6a8b29e"
Jul 24 17:49:55.797679 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:55.797663    2578 scope.go:117] "RemoveContainer" containerID="d88d3c3abf1d126fddccc481d97750465061e2aca2284c8f2e6541d9b6a8b29e"
Jul 24 17:49:55.797941 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:49:55.797922    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d88d3c3abf1d126fddccc481d97750465061e2aca2284c8f2e6541d9b6a8b29e\": container with ID starting with d88d3c3abf1d126fddccc481d97750465061e2aca2284c8f2e6541d9b6a8b29e not found: ID does not exist" containerID="d88d3c3abf1d126fddccc481d97750465061e2aca2284c8f2e6541d9b6a8b29e"
Jul 24 17:49:55.798020 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:55.797958    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d88d3c3abf1d126fddccc481d97750465061e2aca2284c8f2e6541d9b6a8b29e"} err="failed to get container status \"d88d3c3abf1d126fddccc481d97750465061e2aca2284c8f2e6541d9b6a8b29e\": rpc error: code = NotFound desc = could not find container \"d88d3c3abf1d126fddccc481d97750465061e2aca2284c8f2e6541d9b6a8b29e\": container with ID starting with d88d3c3abf1d126fddccc481d97750465061e2aca2284c8f2e6541d9b6a8b29e not found: ID does not exist"
Jul 24 17:49:55.811887 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:55.811862    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9"]
Jul 24 17:49:55.815343 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:55.815321    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-xfbk9"]
Jul 24 17:49:55.967092 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:55.967059    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="561c4b23-4286-4f3e-948d-a6aa35d020f2" path="/var/lib/kubelet/pods/561c4b23-4286-4f3e-948d-a6aa35d020f2/volumes"
Jul 24 17:49:59.240279 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:59.240244    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:59.330037 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:49:59.330003    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:05.607351 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:05.607317    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:05.707102 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:05.707069    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:05.769548 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:05.769515    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:13.444980 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.444942    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:13.445599 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.445575    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-n6bsn"]
Jul 24 17:50:13.445973 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.445956    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="561c4b23-4286-4f3e-948d-a6aa35d020f2" containerName="manager"
Jul 24 17:50:13.446062 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.445974    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="561c4b23-4286-4f3e-948d-a6aa35d020f2" containerName="manager"
Jul 24 17:50:13.446121 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.446087    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="561c4b23-4286-4f3e-948d-a6aa35d020f2" containerName="manager"
Jul 24 17:50:13.448562 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.448544    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-n6bsn"
Jul 24 17:50:13.451043 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.451027    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kfgrn\""
Jul 24 17:50:13.451149 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.451065    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 24 17:50:13.457546 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.457524    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-n6bsn"]
Jul 24 17:50:13.541281 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.541243    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-n6bsn"]
Jul 24 17:50:13.544188 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.544155    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:13.600119 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.600090    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:13.619295 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.619268    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ec1e21f0-a781-4058-857f-d1eb6823e4a4-config-file\") pod \"limitador-limitador-859c4d96d6-n6bsn\" (UID: \"ec1e21f0-a781-4058-857f-d1eb6823e4a4\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-n6bsn"
Jul 24 17:50:13.619414 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.619332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjjl2\" (UniqueName: \"kubernetes.io/projected/ec1e21f0-a781-4058-857f-d1eb6823e4a4-kube-api-access-rjjl2\") pod \"limitador-limitador-859c4d96d6-n6bsn\" (UID: \"ec1e21f0-a781-4058-857f-d1eb6823e4a4\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-n6bsn"
Jul 24 17:50:13.720588 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.720500    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjjl2\" (UniqueName: \"kubernetes.io/projected/ec1e21f0-a781-4058-857f-d1eb6823e4a4-kube-api-access-rjjl2\") pod \"limitador-limitador-859c4d96d6-n6bsn\" (UID: \"ec1e21f0-a781-4058-857f-d1eb6823e4a4\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-n6bsn"
Jul 24 17:50:13.720588 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.720569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ec1e21f0-a781-4058-857f-d1eb6823e4a4-config-file\") pod \"limitador-limitador-859c4d96d6-n6bsn\" (UID: \"ec1e21f0-a781-4058-857f-d1eb6823e4a4\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-n6bsn"
Jul 24 17:50:13.721194 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.721176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ec1e21f0-a781-4058-857f-d1eb6823e4a4-config-file\") pod \"limitador-limitador-859c4d96d6-n6bsn\" (UID: \"ec1e21f0-a781-4058-857f-d1eb6823e4a4\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-n6bsn"
Jul 24 17:50:13.729796 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.729764    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjjl2\" (UniqueName: \"kubernetes.io/projected/ec1e21f0-a781-4058-857f-d1eb6823e4a4-kube-api-access-rjjl2\") pod \"limitador-limitador-859c4d96d6-n6bsn\" (UID: \"ec1e21f0-a781-4058-857f-d1eb6823e4a4\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-n6bsn"
Jul 24 17:50:13.758811 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.758767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-n6bsn"
Jul 24 17:50:13.895476 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:13.895445    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-n6bsn"]
Jul 24 17:50:13.898309 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:50:13.898285    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1e21f0_a781_4058_857f_d1eb6823e4a4.slice/crio-77d7db764f156ce6a3fde61877f089fe146406ff3b9162d89f7ca19b56552edd WatchSource:0}: Error finding container 77d7db764f156ce6a3fde61877f089fe146406ff3b9162d89f7ca19b56552edd: Status 404 returned error can't find the container with id 77d7db764f156ce6a3fde61877f089fe146406ff3b9162d89f7ca19b56552edd
Jul 24 17:50:14.302039 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:14.302003    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:14.402485 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:14.402459    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:14.874757 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:14.874723    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-n6bsn" event={"ID":"ec1e21f0-a781-4058-857f-d1eb6823e4a4","Type":"ContainerStarted","Data":"77d7db764f156ce6a3fde61877f089fe146406ff3b9162d89f7ca19b56552edd"}
Jul 24 17:50:19.899464 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:19.899426    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-n6bsn" event={"ID":"ec1e21f0-a781-4058-857f-d1eb6823e4a4","Type":"ContainerStarted","Data":"c4220b7a9195c1cca084cf7c07ab9f4f798ad3d9b7163450175f1c0690c95ef3"}
Jul 24 17:50:19.899936 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:19.899579    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-n6bsn"
Jul 24 17:50:19.916218 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:19.916148    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-n6bsn" podStartSLOduration=1.479276017 podStartE2EDuration="6.916134716s" totalImagesPullingTime="5.436858699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:50:19.915874733 +0000 UTC m=+442.439233230" watchObservedRunningTime="2026-07-24 17:50:19.916134716 +0000 UTC m=+442.439493194"
Jul 24 17:50:26.680050 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:26.680007    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-n6bsn"]
Jul 24 17:50:26.680554 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:26.680268    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-n6bsn" podUID="ec1e21f0-a781-4058-857f-d1eb6823e4a4" containerName="limitador" containerID="cri-o://c4220b7a9195c1cca084cf7c07ab9f4f798ad3d9b7163450175f1c0690c95ef3" gracePeriod=30
Jul 24 17:50:26.682477 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:26.682453    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-n6bsn"
Jul 24 17:50:27.221431 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:27.221408    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-n6bsn"
Jul 24 17:50:27.243753 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:27.243691    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjjl2\" (UniqueName: \"kubernetes.io/projected/ec1e21f0-a781-4058-857f-d1eb6823e4a4-kube-api-access-rjjl2\") pod \"ec1e21f0-a781-4058-857f-d1eb6823e4a4\" (UID: \"ec1e21f0-a781-4058-857f-d1eb6823e4a4\") "
Jul 24 17:50:27.243753 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:27.243739    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ec1e21f0-a781-4058-857f-d1eb6823e4a4-config-file\") pod \"ec1e21f0-a781-4058-857f-d1eb6823e4a4\" (UID: \"ec1e21f0-a781-4058-857f-d1eb6823e4a4\") "
Jul 24 17:50:27.244086 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:27.244054    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1e21f0-a781-4058-857f-d1eb6823e4a4-config-file" (OuterVolumeSpecName: "config-file") pod "ec1e21f0-a781-4058-857f-d1eb6823e4a4" (UID: "ec1e21f0-a781-4058-857f-d1eb6823e4a4"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:50:27.245731 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:27.245711    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1e21f0-a781-4058-857f-d1eb6823e4a4-kube-api-access-rjjl2" (OuterVolumeSpecName: "kube-api-access-rjjl2") pod "ec1e21f0-a781-4058-857f-d1eb6823e4a4" (UID: "ec1e21f0-a781-4058-857f-d1eb6823e4a4"). InnerVolumeSpecName "kube-api-access-rjjl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:50:27.344739 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:27.344692    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjjl2\" (UniqueName: \"kubernetes.io/projected/ec1e21f0-a781-4058-857f-d1eb6823e4a4-kube-api-access-rjjl2\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:50:27.344739 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:27.344735    2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ec1e21f0-a781-4058-857f-d1eb6823e4a4-config-file\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:50:27.935749 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:27.935716    2578 generic.go:356] "Generic (PLEG): container finished" podID="ec1e21f0-a781-4058-857f-d1eb6823e4a4" containerID="c4220b7a9195c1cca084cf7c07ab9f4f798ad3d9b7163450175f1c0690c95ef3" exitCode=0
Jul 24 17:50:27.936153 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:27.935756    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-n6bsn"
Jul 24 17:50:27.936153 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:27.935761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-n6bsn" event={"ID":"ec1e21f0-a781-4058-857f-d1eb6823e4a4","Type":"ContainerDied","Data":"c4220b7a9195c1cca084cf7c07ab9f4f798ad3d9b7163450175f1c0690c95ef3"}
Jul 24 17:50:27.936153 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:27.935832    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-n6bsn" event={"ID":"ec1e21f0-a781-4058-857f-d1eb6823e4a4","Type":"ContainerDied","Data":"77d7db764f156ce6a3fde61877f089fe146406ff3b9162d89f7ca19b56552edd"}
Jul 24 17:50:27.936153 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:27.935854    2578 scope.go:117] "RemoveContainer" containerID="c4220b7a9195c1cca084cf7c07ab9f4f798ad3d9b7163450175f1c0690c95ef3"
Jul 24 17:50:27.949954 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:27.949935    2578 scope.go:117] "RemoveContainer" containerID="c4220b7a9195c1cca084cf7c07ab9f4f798ad3d9b7163450175f1c0690c95ef3"
Jul 24 17:50:27.950228 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:50:27.950210    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4220b7a9195c1cca084cf7c07ab9f4f798ad3d9b7163450175f1c0690c95ef3\": container with ID starting with c4220b7a9195c1cca084cf7c07ab9f4f798ad3d9b7163450175f1c0690c95ef3 not found: ID does not exist" containerID="c4220b7a9195c1cca084cf7c07ab9f4f798ad3d9b7163450175f1c0690c95ef3"
Jul 24 17:50:27.950280 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:27.950236    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4220b7a9195c1cca084cf7c07ab9f4f798ad3d9b7163450175f1c0690c95ef3"} err="failed to get container status \"c4220b7a9195c1cca084cf7c07ab9f4f798ad3d9b7163450175f1c0690c95ef3\": rpc error: code = NotFound desc = could not find container \"c4220b7a9195c1cca084cf7c07ab9f4f798ad3d9b7163450175f1c0690c95ef3\": container with ID starting with c4220b7a9195c1cca084cf7c07ab9f4f798ad3d9b7163450175f1c0690c95ef3 not found: ID does not exist"
Jul 24 17:50:27.959941 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:27.959874    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-n6bsn"]
Jul 24 17:50:27.967163 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:27.967140    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-n6bsn"]
Jul 24 17:50:28.087147 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.087111    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:28.183281 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.183253    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:28.587467 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.587437    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-z5c7x"]
Jul 24 17:50:28.587946 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.587927    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec1e21f0-a781-4058-857f-d1eb6823e4a4" containerName="limitador"
Jul 24 17:50:28.588050 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.587948    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec1e21f0-a781-4058-857f-d1eb6823e4a4" containerName="limitador"
Jul 24 17:50:28.588050 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.588026    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec1e21f0-a781-4058-857f-d1eb6823e4a4" containerName="limitador"
Jul 24 17:50:28.590556 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.590531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5c7x"
Jul 24 17:50:28.593838 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.593817    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 17:50:28.593960 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.593827    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 24 17:50:28.594800 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.594761    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 17:50:28.594904 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.594763    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-tnwxm\""
Jul 24 17:50:28.599992 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.599817    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-z5c7x"]
Jul 24 17:50:28.630418 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.630364    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:28.655228 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.655193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3805b371-4b1d-427e-bd1e-1568461b89e4-data\") pod \"postgres-69fb4594cf-z5c7x\" (UID: \"3805b371-4b1d-427e-bd1e-1568461b89e4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5c7x"
Jul 24 17:50:28.655382 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.655244    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6ffg\" (UniqueName: \"kubernetes.io/projected/3805b371-4b1d-427e-bd1e-1568461b89e4-kube-api-access-h6ffg\") pod \"postgres-69fb4594cf-z5c7x\" (UID: \"3805b371-4b1d-427e-bd1e-1568461b89e4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5c7x"
Jul 24 17:50:28.755836 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.755805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3805b371-4b1d-427e-bd1e-1568461b89e4-data\") pod \"postgres-69fb4594cf-z5c7x\" (UID: \"3805b371-4b1d-427e-bd1e-1568461b89e4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5c7x"
Jul 24 17:50:28.756016 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.755855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6ffg\" (UniqueName: \"kubernetes.io/projected/3805b371-4b1d-427e-bd1e-1568461b89e4-kube-api-access-h6ffg\") pod \"postgres-69fb4594cf-z5c7x\" (UID: \"3805b371-4b1d-427e-bd1e-1568461b89e4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5c7x"
Jul 24 17:50:28.756185 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.756167    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3805b371-4b1d-427e-bd1e-1568461b89e4-data\") pod \"postgres-69fb4594cf-z5c7x\" (UID: \"3805b371-4b1d-427e-bd1e-1568461b89e4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5c7x"
Jul 24 17:50:28.767922 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.767895    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6ffg\" (UniqueName: \"kubernetes.io/projected/3805b371-4b1d-427e-bd1e-1568461b89e4-kube-api-access-h6ffg\") pod \"postgres-69fb4594cf-z5c7x\" (UID: \"3805b371-4b1d-427e-bd1e-1568461b89e4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5c7x"
Jul 24 17:50:28.903316 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:28.903213    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5c7x"
Jul 24 17:50:29.035994 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:29.035966    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-z5c7x"]
Jul 24 17:50:29.038375 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:50:29.038346    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3805b371_4b1d_427e_bd1e_1568461b89e4.slice/crio-56221d4822071a2beefa7ba114005f0d5cbe99650669193cb4b13ff005d4555c WatchSource:0}: Error finding container 56221d4822071a2beefa7ba114005f0d5cbe99650669193cb4b13ff005d4555c: Status 404 returned error can't find the container with id 56221d4822071a2beefa7ba114005f0d5cbe99650669193cb4b13ff005d4555c
Jul 24 17:50:29.950257 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:29.950186    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5c7x" event={"ID":"3805b371-4b1d-427e-bd1e-1568461b89e4","Type":"ContainerStarted","Data":"56221d4822071a2beefa7ba114005f0d5cbe99650669193cb4b13ff005d4555c"}
Jul 24 17:50:29.969920 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:29.969888    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec1e21f0-a781-4058-857f-d1eb6823e4a4" path="/var/lib/kubelet/pods/ec1e21f0-a781-4058-857f-d1eb6823e4a4/volumes"
Jul 24 17:50:33.976260 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:33.976173    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5c7x" event={"ID":"3805b371-4b1d-427e-bd1e-1568461b89e4","Type":"ContainerStarted","Data":"4eecfaa7cf2ec816e5c69c5640e1ddf85320f5c7fb4e03ee393a7f8db5ea6b5f"}
Jul 24 17:50:33.976260 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:33.976240    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5c7x"
Jul 24 17:50:33.994421 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:33.994372    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5c7x" podStartSLOduration=1.3387919 podStartE2EDuration="5.994356872s" totalImagesPullingTime="4.655564972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:50:33.99276419 +0000 UTC m=+456.516122669" watchObservedRunningTime="2026-07-24 17:50:33.994356872 +0000 UTC m=+456.517715350"
Jul 24 17:50:40.008252 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:40.008214    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5c7x"
Jul 24 17:50:40.351841 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:40.351764    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:40.620000 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:40.619914    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:40.704220 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:40.704185    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-xsdjn"]
Jul 24 17:50:40.707752 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:40.707730    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-xsdjn"
Jul 24 17:50:40.711248 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:40.711227    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mxjvz\""
Jul 24 17:50:40.711248 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:40.711239    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 24 17:50:40.715174 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:40.715152    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-xsdjn"]
Jul 24 17:50:40.761989 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:40.761959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a444f78a-bbf3-48a5-8714-31b0debbac85-tls-cert\") pod \"authorino-687bd45585-xsdjn\" (UID: \"a444f78a-bbf3-48a5-8714-31b0debbac85\") " pod="kuadrant-system/authorino-687bd45585-xsdjn"
Jul 24 17:50:40.762129 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:40.762010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tk6m\" (UniqueName: \"kubernetes.io/projected/a444f78a-bbf3-48a5-8714-31b0debbac85-kube-api-access-6tk6m\") pod \"authorino-687bd45585-xsdjn\" (UID: \"a444f78a-bbf3-48a5-8714-31b0debbac85\") " pod="kuadrant-system/authorino-687bd45585-xsdjn"
Jul 24 17:50:40.862755 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:40.862723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a444f78a-bbf3-48a5-8714-31b0debbac85-tls-cert\") pod \"authorino-687bd45585-xsdjn\" (UID: \"a444f78a-bbf3-48a5-8714-31b0debbac85\") " pod="kuadrant-system/authorino-687bd45585-xsdjn"
Jul 24 17:50:40.862932 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:40.862812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tk6m\" (UniqueName: \"kubernetes.io/projected/a444f78a-bbf3-48a5-8714-31b0debbac85-kube-api-access-6tk6m\") pod \"authorino-687bd45585-xsdjn\" (UID: \"a444f78a-bbf3-48a5-8714-31b0debbac85\") " pod="kuadrant-system/authorino-687bd45585-xsdjn"
Jul 24 17:50:40.865221 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:40.865202    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a444f78a-bbf3-48a5-8714-31b0debbac85-tls-cert\") pod \"authorino-687bd45585-xsdjn\" (UID: \"a444f78a-bbf3-48a5-8714-31b0debbac85\") " pod="kuadrant-system/authorino-687bd45585-xsdjn"
Jul 24 17:50:40.871181 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:40.871115    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tk6m\" (UniqueName: \"kubernetes.io/projected/a444f78a-bbf3-48a5-8714-31b0debbac85-kube-api-access-6tk6m\") pod \"authorino-687bd45585-xsdjn\" (UID: \"a444f78a-bbf3-48a5-8714-31b0debbac85\") " pod="kuadrant-system/authorino-687bd45585-xsdjn"
Jul 24 17:50:40.876356 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:40.876333    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-xsdjn"]
Jul 24 17:50:40.876584 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:40.876571    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-xsdjn"
Jul 24 17:50:41.005244 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:41.005221    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-xsdjn"]
Jul 24 17:50:41.008013 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:50:41.007987    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda444f78a_bbf3_48a5_8714_31b0debbac85.slice/crio-9cec6a2cdbe47f966b1b1e037b74427bcb7d74da3bb3aef1c3b4d46905f27562 WatchSource:0}: Error finding container 9cec6a2cdbe47f966b1b1e037b74427bcb7d74da3bb3aef1c3b4d46905f27562: Status 404 returned error can't find the container with id 9cec6a2cdbe47f966b1b1e037b74427bcb7d74da3bb3aef1c3b4d46905f27562
Jul 24 17:50:42.012251 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:42.012211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687bd45585-xsdjn" event={"ID":"a444f78a-bbf3-48a5-8714-31b0debbac85","Type":"ContainerStarted","Data":"9cec6a2cdbe47f966b1b1e037b74427bcb7d74da3bb3aef1c3b4d46905f27562"}
Jul 24 17:50:44.023034 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:44.022943    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687bd45585-xsdjn" event={"ID":"a444f78a-bbf3-48a5-8714-31b0debbac85","Type":"ContainerStarted","Data":"7a633681e22b5e7a245053f77d59ade2359b68f222d9a6e45bd7db974937ba92"}
Jul 24 17:50:44.023034 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:44.023004    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-687bd45585-xsdjn" podUID="a444f78a-bbf3-48a5-8714-31b0debbac85" containerName="authorino" containerID="cri-o://7a633681e22b5e7a245053f77d59ade2359b68f222d9a6e45bd7db974937ba92" gracePeriod=30
Jul 24 17:50:44.038386 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:44.038341    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-687bd45585-xsdjn" podStartSLOduration=1.396723924 podStartE2EDuration="4.038326867s" totalImagesPullingTime="2.641602943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:50:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:50:44.037011628 +0000 UTC m=+466.560370106" watchObservedRunningTime="2026-07-24 17:50:44.038326867 +0000 UTC m=+466.561685344"
Jul 24 17:50:44.345025 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:44.344994    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-xsdjn"
Jul 24 17:50:44.395031 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:44.394992    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a444f78a-bbf3-48a5-8714-31b0debbac85-tls-cert\") pod \"a444f78a-bbf3-48a5-8714-31b0debbac85\" (UID: \"a444f78a-bbf3-48a5-8714-31b0debbac85\") "
Jul 24 17:50:44.395225 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:44.395059    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tk6m\" (UniqueName: \"kubernetes.io/projected/a444f78a-bbf3-48a5-8714-31b0debbac85-kube-api-access-6tk6m\") pod \"a444f78a-bbf3-48a5-8714-31b0debbac85\" (UID: \"a444f78a-bbf3-48a5-8714-31b0debbac85\") "
Jul 24 17:50:44.396994 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:44.396966    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a444f78a-bbf3-48a5-8714-31b0debbac85-kube-api-access-6tk6m" (OuterVolumeSpecName: "kube-api-access-6tk6m") pod "a444f78a-bbf3-48a5-8714-31b0debbac85" (UID: "a444f78a-bbf3-48a5-8714-31b0debbac85"). InnerVolumeSpecName "kube-api-access-6tk6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:50:44.404702 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:44.404675    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a444f78a-bbf3-48a5-8714-31b0debbac85-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a444f78a-bbf3-48a5-8714-31b0debbac85" (UID: "a444f78a-bbf3-48a5-8714-31b0debbac85"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:50:44.495724 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:44.495679    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tk6m\" (UniqueName: \"kubernetes.io/projected/a444f78a-bbf3-48a5-8714-31b0debbac85-kube-api-access-6tk6m\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:50:44.495724 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:44.495718    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a444f78a-bbf3-48a5-8714-31b0debbac85-tls-cert\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:50:45.028618 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:45.028535    2578 generic.go:356] "Generic (PLEG): container finished" podID="a444f78a-bbf3-48a5-8714-31b0debbac85" containerID="7a633681e22b5e7a245053f77d59ade2359b68f222d9a6e45bd7db974937ba92" exitCode=0
Jul 24 17:50:45.028618 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:45.028573    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-xsdjn"
Jul 24 17:50:45.029092 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:45.028615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687bd45585-xsdjn" event={"ID":"a444f78a-bbf3-48a5-8714-31b0debbac85","Type":"ContainerDied","Data":"7a633681e22b5e7a245053f77d59ade2359b68f222d9a6e45bd7db974937ba92"}
Jul 24 17:50:45.029092 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:45.028654    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687bd45585-xsdjn" event={"ID":"a444f78a-bbf3-48a5-8714-31b0debbac85","Type":"ContainerDied","Data":"9cec6a2cdbe47f966b1b1e037b74427bcb7d74da3bb3aef1c3b4d46905f27562"}
Jul 24 17:50:45.029092 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:45.028674    2578 scope.go:117] "RemoveContainer" containerID="7a633681e22b5e7a245053f77d59ade2359b68f222d9a6e45bd7db974937ba92"
Jul 24 17:50:45.037631 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:45.037615    2578 scope.go:117] "RemoveContainer" containerID="7a633681e22b5e7a245053f77d59ade2359b68f222d9a6e45bd7db974937ba92"
Jul 24 17:50:45.037871 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:50:45.037852    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a633681e22b5e7a245053f77d59ade2359b68f222d9a6e45bd7db974937ba92\": container with ID starting with 7a633681e22b5e7a245053f77d59ade2359b68f222d9a6e45bd7db974937ba92 not found: ID does not exist" containerID="7a633681e22b5e7a245053f77d59ade2359b68f222d9a6e45bd7db974937ba92"
Jul 24 17:50:45.037925 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:45.037884    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a633681e22b5e7a245053f77d59ade2359b68f222d9a6e45bd7db974937ba92"} err="failed to get container status \"7a633681e22b5e7a245053f77d59ade2359b68f222d9a6e45bd7db974937ba92\": rpc error: code = NotFound desc = could not find container \"7a633681e22b5e7a245053f77d59ade2359b68f222d9a6e45bd7db974937ba92\": container with ID starting with 7a633681e22b5e7a245053f77d59ade2359b68f222d9a6e45bd7db974937ba92 not found: ID does not exist"
Jul 24 17:50:45.050302 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:45.050272    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-xsdjn"]
Jul 24 17:50:45.052271 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:45.052249    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-xsdjn"]
Jul 24 17:50:45.967468 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:45.967434    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a444f78a-bbf3-48a5-8714-31b0debbac85" path="/var/lib/kubelet/pods/a444f78a-bbf3-48a5-8714-31b0debbac85/volumes"
Jul 24 17:50:47.719140 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.719090    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:47.737891 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.737860    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-556dd684c-c48z4"]
Jul 24 17:50:47.738222 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.738210    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a444f78a-bbf3-48a5-8714-31b0debbac85" containerName="authorino"
Jul 24 17:50:47.738279 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.738223    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a444f78a-bbf3-48a5-8714-31b0debbac85" containerName="authorino"
Jul 24 17:50:47.738279 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.738275    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a444f78a-bbf3-48a5-8714-31b0debbac85" containerName="authorino"
Jul 24 17:50:47.741091 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.741072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-556dd684c-c48z4"
Jul 24 17:50:47.744640 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.744617    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 17:50:47.744755 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.744726    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8cxvz\""
Jul 24 17:50:47.744755 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.744735    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 17:50:47.753237 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.753214    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-556dd684c-c48z4"]
Jul 24 17:50:47.818846 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.818819    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:47.824435 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.824409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vslb7\" (UniqueName: \"kubernetes.io/projected/460af130-1e4b-492d-87c4-67b887fb0fd6-kube-api-access-vslb7\") pod \"maas-controller-556dd684c-c48z4\" (UID: \"460af130-1e4b-492d-87c4-67b887fb0fd6\") " pod="opendatahub/maas-controller-556dd684c-c48z4"
Jul 24 17:50:47.824562 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.824540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/460af130-1e4b-492d-87c4-67b887fb0fd6-webhook-cert\") pod \"maas-controller-556dd684c-c48z4\" (UID: \"460af130-1e4b-492d-87c4-67b887fb0fd6\") " pod="opendatahub/maas-controller-556dd684c-c48z4"
Jul 24 17:50:47.919180 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.919142    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:47.925859 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.925832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vslb7\" (UniqueName: \"kubernetes.io/projected/460af130-1e4b-492d-87c4-67b887fb0fd6-kube-api-access-vslb7\") pod \"maas-controller-556dd684c-c48z4\" (UID: \"460af130-1e4b-492d-87c4-67b887fb0fd6\") " pod="opendatahub/maas-controller-556dd684c-c48z4"
Jul 24 17:50:47.926008 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.925898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/460af130-1e4b-492d-87c4-67b887fb0fd6-webhook-cert\") pod \"maas-controller-556dd684c-c48z4\" (UID: \"460af130-1e4b-492d-87c4-67b887fb0fd6\") " pod="opendatahub/maas-controller-556dd684c-c48z4"
Jul 24 17:50:47.926463 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.926442    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bf5764cb5-lz4wz"]
Jul 24 17:50:47.928388 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.928367    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/460af130-1e4b-492d-87c4-67b887fb0fd6-webhook-cert\") pod \"maas-controller-556dd684c-c48z4\" (UID: \"460af130-1e4b-492d-87c4-67b887fb0fd6\") " pod="opendatahub/maas-controller-556dd684c-c48z4"
Jul 24 17:50:47.929859 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.929842    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf5764cb5-lz4wz"
Jul 24 17:50:47.935090 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.935071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vslb7\" (UniqueName: \"kubernetes.io/projected/460af130-1e4b-492d-87c4-67b887fb0fd6-kube-api-access-vslb7\") pod \"maas-controller-556dd684c-c48z4\" (UID: \"460af130-1e4b-492d-87c4-67b887fb0fd6\") " pod="opendatahub/maas-controller-556dd684c-c48z4"
Jul 24 17:50:47.940948 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.940920    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bf5764cb5-lz4wz"]
Jul 24 17:50:47.980697 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:47.980619    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:48.027114 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:48.027078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55c5941b-f87c-4d4c-83d9-c400ea3d482e-webhook-cert\") pod \"maas-controller-bf5764cb5-lz4wz\" (UID: \"55c5941b-f87c-4d4c-83d9-c400ea3d482e\") " pod="opendatahub/maas-controller-bf5764cb5-lz4wz"
Jul 24 17:50:48.027273 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:48.027161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfthc\" (UniqueName: \"kubernetes.io/projected/55c5941b-f87c-4d4c-83d9-c400ea3d482e-kube-api-access-lfthc\") pod \"maas-controller-bf5764cb5-lz4wz\" (UID: \"55c5941b-f87c-4d4c-83d9-c400ea3d482e\") " pod="opendatahub/maas-controller-bf5764cb5-lz4wz"
Jul 24 17:50:48.051970 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:48.051942    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-556dd684c-c48z4"
Jul 24 17:50:48.128695 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:48.128432    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55c5941b-f87c-4d4c-83d9-c400ea3d482e-webhook-cert\") pod \"maas-controller-bf5764cb5-lz4wz\" (UID: \"55c5941b-f87c-4d4c-83d9-c400ea3d482e\") " pod="opendatahub/maas-controller-bf5764cb5-lz4wz"
Jul 24 17:50:48.128695 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:48.128533    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfthc\" (UniqueName: \"kubernetes.io/projected/55c5941b-f87c-4d4c-83d9-c400ea3d482e-kube-api-access-lfthc\") pod \"maas-controller-bf5764cb5-lz4wz\" (UID: \"55c5941b-f87c-4d4c-83d9-c400ea3d482e\") " pod="opendatahub/maas-controller-bf5764cb5-lz4wz"
Jul 24 17:50:48.131870 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:48.131843    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55c5941b-f87c-4d4c-83d9-c400ea3d482e-webhook-cert\") pod \"maas-controller-bf5764cb5-lz4wz\" (UID: \"55c5941b-f87c-4d4c-83d9-c400ea3d482e\") " pod="opendatahub/maas-controller-bf5764cb5-lz4wz"
Jul 24 17:50:48.137228 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:48.137206    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfthc\" (UniqueName: \"kubernetes.io/projected/55c5941b-f87c-4d4c-83d9-c400ea3d482e-kube-api-access-lfthc\") pod \"maas-controller-bf5764cb5-lz4wz\" (UID: \"55c5941b-f87c-4d4c-83d9-c400ea3d482e\") " pod="opendatahub/maas-controller-bf5764cb5-lz4wz"
Jul 24 17:50:48.177511 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:48.177486    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-556dd684c-c48z4"]
Jul 24 17:50:48.179430 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:50:48.179396    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460af130_1e4b_492d_87c4_67b887fb0fd6.slice/crio-8a9be7228d1d192a131588eb4cb14762bf6bb86c7af0e472944024491a592c6e WatchSource:0}: Error finding container 8a9be7228d1d192a131588eb4cb14762bf6bb86c7af0e472944024491a592c6e: Status 404 returned error can't find the container with id 8a9be7228d1d192a131588eb4cb14762bf6bb86c7af0e472944024491a592c6e
Jul 24 17:50:48.252302 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:48.252221    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf5764cb5-lz4wz"
Jul 24 17:50:48.381455 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:48.381432    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bf5764cb5-lz4wz"]
Jul 24 17:50:48.383714 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:50:48.383685    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c5941b_f87c_4d4c_83d9_c400ea3d482e.slice/crio-3dab2c280b51f843ff5b1fb411ae2af0bea8d38b5573934f873288af48201232 WatchSource:0}: Error finding container 3dab2c280b51f843ff5b1fb411ae2af0bea8d38b5573934f873288af48201232: Status 404 returned error can't find the container with id 3dab2c280b51f843ff5b1fb411ae2af0bea8d38b5573934f873288af48201232
Jul 24 17:50:49.051561 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:49.051516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf5764cb5-lz4wz" event={"ID":"55c5941b-f87c-4d4c-83d9-c400ea3d482e","Type":"ContainerStarted","Data":"3dab2c280b51f843ff5b1fb411ae2af0bea8d38b5573934f873288af48201232"}
Jul 24 17:50:49.054251 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:49.053589    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-556dd684c-c48z4" event={"ID":"460af130-1e4b-492d-87c4-67b887fb0fd6","Type":"ContainerStarted","Data":"8a9be7228d1d192a131588eb4cb14762bf6bb86c7af0e472944024491a592c6e"}
Jul 24 17:50:50.921172 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:50.921138    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:51.046431 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:51.046398    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:51.067095 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:51.067063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf5764cb5-lz4wz" event={"ID":"55c5941b-f87c-4d4c-83d9-c400ea3d482e","Type":"ContainerStarted","Data":"5d873b49dd5d1b6090310d56e519d9748dd40c8855790fec508d83b595bf5c20"}
Jul 24 17:50:51.067250 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:51.067129    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bf5764cb5-lz4wz"
Jul 24 17:50:51.068450 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:51.068426    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-556dd684c-c48z4" event={"ID":"460af130-1e4b-492d-87c4-67b887fb0fd6","Type":"ContainerStarted","Data":"5751d22191a7605895c65fc97187d3917892d3c8316c5d3654a3768c03181cd3"}
Jul 24 17:50:51.068566 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:51.068498    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-556dd684c-c48z4"
Jul 24 17:50:51.095066 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:51.095020    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bf5764cb5-lz4wz" podStartSLOduration=1.747047946 podStartE2EDuration="4.095005558s" totalImagesPullingTime="2.347957612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:50:51.093526142 +0000 UTC m=+473.616884621" watchObservedRunningTime="2026-07-24 17:50:51.095005558 +0000 UTC m=+473.618364043"
Jul 24 17:50:51.115713 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:50:51.115664    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-556dd684c-c48z4" podStartSLOduration=1.556658697 podStartE2EDuration="4.115649183s" totalImagesPullingTime="2.558990486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:50:51.115423543 +0000 UTC m=+473.638782022" watchObservedRunningTime="2026-07-24 17:50:51.115649183 +0000 UTC m=+473.639007661"
Jul 24 17:51:02.075390 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:02.075357    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-556dd684c-c48z4"
Jul 24 17:51:02.075868 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:02.075443    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bf5764cb5-lz4wz"
Jul 24 17:51:02.143023 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:02.142990    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-556dd684c-c48z4"]
Jul 24 17:51:02.143745 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:02.143215    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-556dd684c-c48z4" podUID="460af130-1e4b-492d-87c4-67b887fb0fd6" containerName="manager" containerID="cri-o://5751d22191a7605895c65fc97187d3917892d3c8316c5d3654a3768c03181cd3" gracePeriod=30
Jul 24 17:51:02.387621 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:02.387592    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-556dd684c-c48z4"
Jul 24 17:51:02.457090 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:02.457063    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vslb7\" (UniqueName: \"kubernetes.io/projected/460af130-1e4b-492d-87c4-67b887fb0fd6-kube-api-access-vslb7\") pod \"460af130-1e4b-492d-87c4-67b887fb0fd6\" (UID: \"460af130-1e4b-492d-87c4-67b887fb0fd6\") "
Jul 24 17:51:02.457090 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:02.457095    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/460af130-1e4b-492d-87c4-67b887fb0fd6-webhook-cert\") pod \"460af130-1e4b-492d-87c4-67b887fb0fd6\" (UID: \"460af130-1e4b-492d-87c4-67b887fb0fd6\") "
Jul 24 17:51:02.459161 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:02.459128    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460af130-1e4b-492d-87c4-67b887fb0fd6-kube-api-access-vslb7" (OuterVolumeSpecName: "kube-api-access-vslb7") pod "460af130-1e4b-492d-87c4-67b887fb0fd6" (UID: "460af130-1e4b-492d-87c4-67b887fb0fd6"). InnerVolumeSpecName "kube-api-access-vslb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:51:02.459274 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:02.459231    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/460af130-1e4b-492d-87c4-67b887fb0fd6-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "460af130-1e4b-492d-87c4-67b887fb0fd6" (UID: "460af130-1e4b-492d-87c4-67b887fb0fd6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:51:02.558472 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:02.558437    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vslb7\" (UniqueName: \"kubernetes.io/projected/460af130-1e4b-492d-87c4-67b887fb0fd6-kube-api-access-vslb7\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:51:02.558472 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:02.558466    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/460af130-1e4b-492d-87c4-67b887fb0fd6-webhook-cert\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:51:03.124909 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:03.124873    2578 generic.go:356] "Generic (PLEG): container finished" podID="460af130-1e4b-492d-87c4-67b887fb0fd6" containerID="5751d22191a7605895c65fc97187d3917892d3c8316c5d3654a3768c03181cd3" exitCode=0
Jul 24 17:51:03.125364 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:03.124921    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-556dd684c-c48z4"
Jul 24 17:51:03.125364 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:03.124933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-556dd684c-c48z4" event={"ID":"460af130-1e4b-492d-87c4-67b887fb0fd6","Type":"ContainerDied","Data":"5751d22191a7605895c65fc97187d3917892d3c8316c5d3654a3768c03181cd3"}
Jul 24 17:51:03.125364 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:03.124985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-556dd684c-c48z4" event={"ID":"460af130-1e4b-492d-87c4-67b887fb0fd6","Type":"ContainerDied","Data":"8a9be7228d1d192a131588eb4cb14762bf6bb86c7af0e472944024491a592c6e"}
Jul 24 17:51:03.125364 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:03.125015    2578 scope.go:117] "RemoveContainer" containerID="5751d22191a7605895c65fc97187d3917892d3c8316c5d3654a3768c03181cd3"
Jul 24 17:51:03.133645 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:03.133623    2578 scope.go:117] "RemoveContainer" containerID="5751d22191a7605895c65fc97187d3917892d3c8316c5d3654a3768c03181cd3"
Jul 24 17:51:03.133910 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:51:03.133891    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5751d22191a7605895c65fc97187d3917892d3c8316c5d3654a3768c03181cd3\": container with ID starting with 5751d22191a7605895c65fc97187d3917892d3c8316c5d3654a3768c03181cd3 not found: ID does not exist" containerID="5751d22191a7605895c65fc97187d3917892d3c8316c5d3654a3768c03181cd3"
Jul 24 17:51:03.133969 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:03.133919    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5751d22191a7605895c65fc97187d3917892d3c8316c5d3654a3768c03181cd3"} err="failed to get container status \"5751d22191a7605895c65fc97187d3917892d3c8316c5d3654a3768c03181cd3\": rpc error: code = NotFound desc = could not find container \"5751d22191a7605895c65fc97187d3917892d3c8316c5d3654a3768c03181cd3\": container with ID starting with 5751d22191a7605895c65fc97187d3917892d3c8316c5d3654a3768c03181cd3 not found: ID does not exist"
Jul 24 17:51:03.146424 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:03.146397    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-556dd684c-c48z4"]
Jul 24 17:51:03.152262 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:03.152243    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-556dd684c-c48z4"]
Jul 24 17:51:03.967041 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:03.967013    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="460af130-1e4b-492d-87c4-67b887fb0fd6" path="/var/lib/kubelet/pods/460af130-1e4b-492d-87c4-67b887fb0fd6/volumes"
Jul 24 17:51:21.892667 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:21.892633    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:51:21.987521 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:21.987484    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:51:22.594870 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:22.594839    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:51:22.610298 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:22.610265    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb"]
Jul 24 17:51:22.610660 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:22.610647    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="460af130-1e4b-492d-87c4-67b887fb0fd6" containerName="manager"
Jul 24 17:51:22.610707 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:22.610662    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="460af130-1e4b-492d-87c4-67b887fb0fd6" containerName="manager"
Jul 24 17:51:22.610742 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:22.610730    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="460af130-1e4b-492d-87c4-67b887fb0fd6" containerName="manager"
Jul 24 17:51:22.616498 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:22.616475    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb"
Jul 24 17:51:22.619868 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:22.619837    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7pptq\""
Jul 24 17:51:22.622048 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:22.622027    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 24 17:51:22.629250 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:22.629229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3-maas-api-tls\") pod \"maas-api-dc545c9bc-5h9hb\" (UID: \"58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3\") " pod="odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb"
Jul 24 17:51:22.629343 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:22.629258    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3-openshift-service-ca\") pod \"maas-api-dc545c9bc-5h9hb\" (UID: \"58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3\") " pod="odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb"
Jul 24 17:51:22.629343 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:22.629282    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwjhn\" (UniqueName: \"kubernetes.io/projected/58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3-kube-api-access-fwjhn\") pod \"maas-api-dc545c9bc-5h9hb\" (UID: \"58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3\") " pod="odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb"
Jul 24 17:51:22.641541 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:22.641520    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb"]
Jul 24 17:51:22.730145 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:22.730110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3-maas-api-tls\") pod \"maas-api-dc545c9bc-5h9hb\" (UID: \"58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3\") " pod="odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb"
Jul 24 17:51:22.730145 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:22.730156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3-openshift-service-ca\") pod \"maas-api-dc545c9bc-5h9hb\" (UID: \"58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3\") " pod="odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb"
Jul 24 17:51:22.730352 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:22.730181    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwjhn\" (UniqueName: \"kubernetes.io/projected/58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3-kube-api-access-fwjhn\") pod \"maas-api-dc545c9bc-5h9hb\" (UID: \"58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3\") " pod="odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb"
Jul 24 17:51:22.730352 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:51:22.730278    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Jul 24 17:51:22.730422 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:51:22.730366    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3-maas-api-tls podName:58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3 nodeName:}" failed. No retries permitted until 2026-07-24 17:51:23.230345384 +0000 UTC m=+505.753703853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3-maas-api-tls") pod "maas-api-dc545c9bc-5h9hb" (UID: "58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3") : secret "maas-api-serving-cert" not found
Jul 24 17:51:22.730863 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:22.730847    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3-openshift-service-ca\") pod \"maas-api-dc545c9bc-5h9hb\" (UID: \"58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3\") " pod="odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb"
Jul 24 17:51:22.742041 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:22.742014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwjhn\" (UniqueName: \"kubernetes.io/projected/58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3-kube-api-access-fwjhn\") pod \"maas-api-dc545c9bc-5h9hb\" (UID: \"58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3\") " pod="odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb"
Jul 24 17:51:22.957391 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:22.957310    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:51:23.057601 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:23.057569    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:51:23.159636 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:23.159599    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:51:23.167591 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:23.167559    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-9g7th"]
Jul 24 17:51:23.171822 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:23.171804    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-9g7th"
Jul 24 17:51:23.174460 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:23.174440    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 24 17:51:23.184290 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:23.184269    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-9g7th"]
Jul 24 17:51:23.234127 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:23.234049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992231c7-b15d-4590-a3d5-615ed717052b-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-9g7th\" (UID: \"992231c7-b15d-4590-a3d5-615ed717052b\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9g7th"
Jul 24 17:51:23.234127 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:23.234089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c8gj\" (UniqueName: \"kubernetes.io/projected/992231c7-b15d-4590-a3d5-615ed717052b-kube-api-access-2c8gj\") pod \"payload-processing-5cc6cd444b-9g7th\" (UID: \"992231c7-b15d-4590-a3d5-615ed717052b\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9g7th"
Jul 24 17:51:23.234127 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:23.234118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3-maas-api-tls\") pod \"maas-api-dc545c9bc-5h9hb\" (UID: \"58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3\") " pod="odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb"
Jul 24 17:51:23.236557 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:23.236535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3-maas-api-tls\") pod \"maas-api-dc545c9bc-5h9hb\" (UID: \"58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3\") " pod="odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb"
Jul 24 17:51:23.249322 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:23.249290    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:51:23.335232 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:23.335189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992231c7-b15d-4590-a3d5-615ed717052b-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-9g7th\" (UID: \"992231c7-b15d-4590-a3d5-615ed717052b\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9g7th"
Jul 24 17:51:23.335413 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:23.335254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8gj\" (UniqueName: \"kubernetes.io/projected/992231c7-b15d-4590-a3d5-615ed717052b-kube-api-access-2c8gj\") pod \"payload-processing-5cc6cd444b-9g7th\" (UID: \"992231c7-b15d-4590-a3d5-615ed717052b\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9g7th"
Jul 24 17:51:23.335907 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:23.335887    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992231c7-b15d-4590-a3d5-615ed717052b-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-9g7th\" (UID: \"992231c7-b15d-4590-a3d5-615ed717052b\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9g7th"
Jul 24 17:51:23.343897 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:23.343875    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c8gj\" (UniqueName: \"kubernetes.io/projected/992231c7-b15d-4590-a3d5-615ed717052b-kube-api-access-2c8gj\") pod \"payload-processing-5cc6cd444b-9g7th\" (UID: \"992231c7-b15d-4590-a3d5-615ed717052b\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9g7th"
Jul 24 17:51:23.483717 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:23.483682    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-9g7th"
Jul 24 17:51:23.526378 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:23.526345    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb"
Jul 24 17:51:23.625219 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:23.625087    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-9g7th"]
Jul 24 17:51:23.684480 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:23.684445    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb"]
Jul 24 17:51:23.686613 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:51:23.686583    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b31e7d_f84a_4bba_8a35_cdb23b3ad3c3.slice/crio-a01bf7cf57e3cacb1693bfe8bf7091fc44a11968022d6a91da2fdc46feabafae WatchSource:0}: Error finding container a01bf7cf57e3cacb1693bfe8bf7091fc44a11968022d6a91da2fdc46feabafae: Status 404 returned error can't find the container with id a01bf7cf57e3cacb1693bfe8bf7091fc44a11968022d6a91da2fdc46feabafae
Jul 24 17:51:24.223047 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:24.222982    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-9g7th" event={"ID":"992231c7-b15d-4590-a3d5-615ed717052b","Type":"ContainerStarted","Data":"c5c32f4e0fd062ec59fe6b5a940747438118a48e93cac532560b455e2242ce34"}
Jul 24 17:51:24.224649 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:24.224597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb" event={"ID":"58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3","Type":"ContainerStarted","Data":"a01bf7cf57e3cacb1693bfe8bf7091fc44a11968022d6a91da2fdc46feabafae"}
Jul 24 17:51:27.244168 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:27.244121    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb" event={"ID":"58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3","Type":"ContainerStarted","Data":"97b7d62f0aae3d560c2a87bfdfece8961c753481a1495a57ee798b9f01117f45"}
Jul 24 17:51:27.244638 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:27.244190    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb"
Jul 24 17:51:27.245698 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:27.245670    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-9g7th" event={"ID":"992231c7-b15d-4590-a3d5-615ed717052b","Type":"ContainerStarted","Data":"5837302237f9f5f98b08c466587eec3ec2fada011c60f96a9bd5bcf199d266c3"}
Jul 24 17:51:27.245852 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:27.245832    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-9g7th"
Jul 24 17:51:27.265871 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:27.265831    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb" podStartSLOduration=2.7649068530000003 podStartE2EDuration="5.265819219s" totalImagesPullingTime="2.500912366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:51:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:51:27.264989007 +0000 UTC m=+509.788347488" watchObservedRunningTime="2026-07-24 17:51:27.265819219 +0000 UTC m=+509.789177697"
Jul 24 17:51:27.287169 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:27.287126    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-9g7th" podStartSLOduration=1.686229867 podStartE2EDuration="4.287113438s" totalImagesPullingTime="2.600883571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:51:27.285749334 +0000 UTC m=+509.809107815" watchObservedRunningTime="2026-07-24 17:51:27.287113438 +0000 UTC m=+509.810471956"
Jul 24 17:51:33.255885 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:33.255854    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-dc545c9bc-5h9hb"
Jul 24 17:51:34.391540 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:34.391504    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:51:34.532558 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:34.532516    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:51:38.252462 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:38.252430    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-9g7th"
Jul 24 17:51:45.269865 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:45.269830    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:51:59.690211 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:51:59.690179    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:01.101991 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:01.101947    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:14.050093 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.050053    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"]
Jul 24 17:52:14.055741 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.055720    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.059353 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.059329    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vxck9\""
Jul 24 17:52:14.059353 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.059347    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 24 17:52:14.059548 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.059445    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 24 17:52:14.059548 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.059503    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 24 17:52:14.063605 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.063586    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"]
Jul 24 17:52:14.075100 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.075080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee74fad-6891-44b0-b0d4-a8009ec58495-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5j9vz\" (UID: \"fee74fad-6891-44b0-b0d4-a8009ec58495\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.075201 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.075106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee74fad-6891-44b0-b0d4-a8009ec58495-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5j9vz\" (UID: \"fee74fad-6891-44b0-b0d4-a8009ec58495\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.075201 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.075151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee74fad-6891-44b0-b0d4-a8009ec58495-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5j9vz\" (UID: \"fee74fad-6891-44b0-b0d4-a8009ec58495\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.075285 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.075194    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb5lg\" (UniqueName: \"kubernetes.io/projected/fee74fad-6891-44b0-b0d4-a8009ec58495-kube-api-access-jb5lg\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5j9vz\" (UID: \"fee74fad-6891-44b0-b0d4-a8009ec58495\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.075285 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.075269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee74fad-6891-44b0-b0d4-a8009ec58495-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5j9vz\" (UID: \"fee74fad-6891-44b0-b0d4-a8009ec58495\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.075361 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.075287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee74fad-6891-44b0-b0d4-a8009ec58495-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5j9vz\" (UID: \"fee74fad-6891-44b0-b0d4-a8009ec58495\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.165765 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.165728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:14.175981 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.175946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee74fad-6891-44b0-b0d4-a8009ec58495-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5j9vz\" (UID: \"fee74fad-6891-44b0-b0d4-a8009ec58495\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.175981 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.175981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb5lg\" (UniqueName: \"kubernetes.io/projected/fee74fad-6891-44b0-b0d4-a8009ec58495-kube-api-access-jb5lg\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5j9vz\" (UID: \"fee74fad-6891-44b0-b0d4-a8009ec58495\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.176208 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.176040    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee74fad-6891-44b0-b0d4-a8009ec58495-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5j9vz\" (UID: \"fee74fad-6891-44b0-b0d4-a8009ec58495\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.176208 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.176068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee74fad-6891-44b0-b0d4-a8009ec58495-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5j9vz\" (UID: \"fee74fad-6891-44b0-b0d4-a8009ec58495\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.176208 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.176115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee74fad-6891-44b0-b0d4-a8009ec58495-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5j9vz\" (UID: \"fee74fad-6891-44b0-b0d4-a8009ec58495\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.176208 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.176139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee74fad-6891-44b0-b0d4-a8009ec58495-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5j9vz\" (UID: \"fee74fad-6891-44b0-b0d4-a8009ec58495\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.176412 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.176395    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee74fad-6891-44b0-b0d4-a8009ec58495-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5j9vz\" (UID: \"fee74fad-6891-44b0-b0d4-a8009ec58495\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.176479 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.176454    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee74fad-6891-44b0-b0d4-a8009ec58495-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5j9vz\" (UID: \"fee74fad-6891-44b0-b0d4-a8009ec58495\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.176572 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.176556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee74fad-6891-44b0-b0d4-a8009ec58495-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5j9vz\" (UID: \"fee74fad-6891-44b0-b0d4-a8009ec58495\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.178340 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.178313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee74fad-6891-44b0-b0d4-a8009ec58495-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5j9vz\" (UID: \"fee74fad-6891-44b0-b0d4-a8009ec58495\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.178703 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.178680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee74fad-6891-44b0-b0d4-a8009ec58495-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5j9vz\" (UID: \"fee74fad-6891-44b0-b0d4-a8009ec58495\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.185533 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.185508    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb5lg\" (UniqueName: \"kubernetes.io/projected/fee74fad-6891-44b0-b0d4-a8009ec58495-kube-api-access-jb5lg\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5j9vz\" (UID: \"fee74fad-6891-44b0-b0d4-a8009ec58495\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.367181 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.367091    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:14.504583 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:14.504557    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"]
Jul 24 17:52:14.507174 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:52:14.507146    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee74fad_6891_44b0_b0d4_a8009ec58495.slice/crio-0730dbf3a9433f375ac28328da21c04b7d78013640cbd8e1f5b2390b484b1b43 WatchSource:0}: Error finding container 0730dbf3a9433f375ac28328da21c04b7d78013640cbd8e1f5b2390b484b1b43: Status 404 returned error can't find the container with id 0730dbf3a9433f375ac28328da21c04b7d78013640cbd8e1f5b2390b484b1b43
Jul 24 17:52:15.469719 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:15.469686    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz" event={"ID":"fee74fad-6891-44b0-b0d4-a8009ec58495","Type":"ContainerStarted","Data":"0730dbf3a9433f375ac28328da21c04b7d78013640cbd8e1f5b2390b484b1b43"}
Jul 24 17:52:16.476625 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:16.476591    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz" event={"ID":"fee74fad-6891-44b0-b0d4-a8009ec58495","Type":"ContainerStarted","Data":"6282722fe9f35d95a75588cc4e678b9300fc879f5b0f56fcb04a4924e112cb7d"}
Jul 24 17:52:20.744369 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.744310    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz" podStartSLOduration=5.746678395 podStartE2EDuration="6.744292332s" totalImagesPullingTime="997.613937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:52:16.499289976 +0000 UTC m=+559.022648455" watchObservedRunningTime="2026-07-24 17:52:20.744292332 +0000 UTC m=+563.267650811"
Jul 24 17:52:20.744743 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.744598    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"]
Jul 24 17:52:20.748236 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.748215    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:20.751502 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.751482    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 24 17:52:20.759296 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.759273    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"]
Jul 24 17:52:20.786928 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.786873    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:20.844266 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.844229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tldkh\" (UniqueName: \"kubernetes.io/projected/fbea9b36-a10d-460c-98fe-7c280e7b02ee-kube-api-access-tldkh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jh2s2\" (UID: \"fbea9b36-a10d-460c-98fe-7c280e7b02ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:20.844431 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.844294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbea9b36-a10d-460c-98fe-7c280e7b02ee-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jh2s2\" (UID: \"fbea9b36-a10d-460c-98fe-7c280e7b02ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:20.844431 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.844334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbea9b36-a10d-460c-98fe-7c280e7b02ee-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jh2s2\" (UID: \"fbea9b36-a10d-460c-98fe-7c280e7b02ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:20.844431 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.844371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbea9b36-a10d-460c-98fe-7c280e7b02ee-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jh2s2\" (UID: \"fbea9b36-a10d-460c-98fe-7c280e7b02ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:20.844431 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.844420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbea9b36-a10d-460c-98fe-7c280e7b02ee-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jh2s2\" (UID: \"fbea9b36-a10d-460c-98fe-7c280e7b02ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:20.844599 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.844463    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbea9b36-a10d-460c-98fe-7c280e7b02ee-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jh2s2\" (UID: \"fbea9b36-a10d-460c-98fe-7c280e7b02ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:20.945441 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.945402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbea9b36-a10d-460c-98fe-7c280e7b02ee-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jh2s2\" (UID: \"fbea9b36-a10d-460c-98fe-7c280e7b02ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:20.945441 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.945442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbea9b36-a10d-460c-98fe-7c280e7b02ee-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jh2s2\" (UID: \"fbea9b36-a10d-460c-98fe-7c280e7b02ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:20.945645 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.945467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbea9b36-a10d-460c-98fe-7c280e7b02ee-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jh2s2\" (UID: \"fbea9b36-a10d-460c-98fe-7c280e7b02ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:20.945645 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.945484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbea9b36-a10d-460c-98fe-7c280e7b02ee-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jh2s2\" (UID: \"fbea9b36-a10d-460c-98fe-7c280e7b02ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:20.945645 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.945530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbea9b36-a10d-460c-98fe-7c280e7b02ee-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jh2s2\" (UID: \"fbea9b36-a10d-460c-98fe-7c280e7b02ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:20.945645 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.945595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tldkh\" (UniqueName: \"kubernetes.io/projected/fbea9b36-a10d-460c-98fe-7c280e7b02ee-kube-api-access-tldkh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jh2s2\" (UID: \"fbea9b36-a10d-460c-98fe-7c280e7b02ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:20.945878 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.945802    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbea9b36-a10d-460c-98fe-7c280e7b02ee-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jh2s2\" (UID: \"fbea9b36-a10d-460c-98fe-7c280e7b02ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:20.946001 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.945979    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbea9b36-a10d-460c-98fe-7c280e7b02ee-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jh2s2\" (UID: \"fbea9b36-a10d-460c-98fe-7c280e7b02ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:20.946058 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.945999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbea9b36-a10d-460c-98fe-7c280e7b02ee-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jh2s2\" (UID: \"fbea9b36-a10d-460c-98fe-7c280e7b02ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:20.947566 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.947540    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbea9b36-a10d-460c-98fe-7c280e7b02ee-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jh2s2\" (UID: \"fbea9b36-a10d-460c-98fe-7c280e7b02ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:20.947925 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.947906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbea9b36-a10d-460c-98fe-7c280e7b02ee-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jh2s2\" (UID: \"fbea9b36-a10d-460c-98fe-7c280e7b02ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:20.956053 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:20.956032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tldkh\" (UniqueName: \"kubernetes.io/projected/fbea9b36-a10d-460c-98fe-7c280e7b02ee-kube-api-access-tldkh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jh2s2\" (UID: \"fbea9b36-a10d-460c-98fe-7c280e7b02ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:21.059573 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:21.059540    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:21.196913 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:21.196882    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"]
Jul 24 17:52:21.200086 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:52:21.200060    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbea9b36_a10d_460c_98fe_7c280e7b02ee.slice/crio-e5e31778c1baa3fe9056005b2b367850bf4b4e52fd0b99e3512e2eaa1f4b6a33 WatchSource:0}: Error finding container e5e31778c1baa3fe9056005b2b367850bf4b4e52fd0b99e3512e2eaa1f4b6a33: Status 404 returned error can't find the container with id e5e31778c1baa3fe9056005b2b367850bf4b4e52fd0b99e3512e2eaa1f4b6a33
Jul 24 17:52:21.502321 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:21.502287    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2" event={"ID":"fbea9b36-a10d-460c-98fe-7c280e7b02ee","Type":"ContainerStarted","Data":"50cf7e382e58ef2c7d06b02183af14df2f3a4cd30e365e16ecade0d23b04f825"}
Jul 24 17:52:21.502483 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:21.502347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2" event={"ID":"fbea9b36-a10d-460c-98fe-7c280e7b02ee","Type":"ContainerStarted","Data":"e5e31778c1baa3fe9056005b2b367850bf4b4e52fd0b99e3512e2eaa1f4b6a33"}
Jul 24 17:52:21.534926 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:21.534851    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2" podStartSLOduration=1.314350094 podStartE2EDuration="1.534830365s" totalImagesPullingTime="220.480271ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:52:21.534023947 +0000 UTC m=+564.057382427" watchObservedRunningTime="2026-07-24 17:52:21.534830365 +0000 UTC m=+564.058188843"
Jul 24 17:52:24.368238 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:24.368202    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:24.368627 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:24.368261    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:24.380858 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:24.380830    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:24.527658 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:24.527630    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5j9vz"
Jul 24 17:52:29.867864 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:29.867826    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:31.060207 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:31.060161    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:31.060573 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:31.060281    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:31.072944 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:31.072918    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:31.561384 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:31.561359    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jh2s2"
Jul 24 17:52:42.780334 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:52:42.780295    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:53:02.453096 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:02.453060    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:53:20.277703 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:20.277588    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:53:34.573199 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:34.573163    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cc64c6669-ghcws"]
Jul 24 17:53:34.576835 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:34.576813    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cc64c6669-ghcws"
Jul 24 17:53:34.580694 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:34.580667    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 24 17:53:34.580694 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:34.580668    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mxjvz\""
Jul 24 17:53:34.587093 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:34.587070    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cc64c6669-ghcws"]
Jul 24 17:53:34.606747 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:34.606720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d8913974-7669-449d-8129-b56ff58193e2-tls-cert\") pod \"authorino-7cc64c6669-ghcws\" (UID: \"d8913974-7669-449d-8129-b56ff58193e2\") " pod="kuadrant-system/authorino-7cc64c6669-ghcws"
Jul 24 17:53:34.606907 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:34.606753    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbh4d\" (UniqueName: \"kubernetes.io/projected/d8913974-7669-449d-8129-b56ff58193e2-kube-api-access-mbh4d\") pod \"authorino-7cc64c6669-ghcws\" (UID: \"d8913974-7669-449d-8129-b56ff58193e2\") " pod="kuadrant-system/authorino-7cc64c6669-ghcws"
Jul 24 17:53:34.708247 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:34.708205    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d8913974-7669-449d-8129-b56ff58193e2-tls-cert\") pod \"authorino-7cc64c6669-ghcws\" (UID: \"d8913974-7669-449d-8129-b56ff58193e2\") " pod="kuadrant-system/authorino-7cc64c6669-ghcws"
Jul 24 17:53:34.708247 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:34.708242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbh4d\" (UniqueName: \"kubernetes.io/projected/d8913974-7669-449d-8129-b56ff58193e2-kube-api-access-mbh4d\") pod \"authorino-7cc64c6669-ghcws\" (UID: \"d8913974-7669-449d-8129-b56ff58193e2\") " pod="kuadrant-system/authorino-7cc64c6669-ghcws"
Jul 24 17:53:34.710814 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:34.710770    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d8913974-7669-449d-8129-b56ff58193e2-tls-cert\") pod \"authorino-7cc64c6669-ghcws\" (UID: \"d8913974-7669-449d-8129-b56ff58193e2\") " pod="kuadrant-system/authorino-7cc64c6669-ghcws"
Jul 24 17:53:34.717719 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:34.717684    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbh4d\" (UniqueName: \"kubernetes.io/projected/d8913974-7669-449d-8129-b56ff58193e2-kube-api-access-mbh4d\") pod \"authorino-7cc64c6669-ghcws\" (UID: \"d8913974-7669-449d-8129-b56ff58193e2\") " pod="kuadrant-system/authorino-7cc64c6669-ghcws"
Jul 24 17:53:34.893111 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:34.893031    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cc64c6669-ghcws"
Jul 24 17:53:35.027216 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:35.027188    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cc64c6669-ghcws"]
Jul 24 17:53:35.029684 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:53:35.029659    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8913974_7669_449d_8129_b56ff58193e2.slice/crio-0b2cdb471bafb75ab5c977e7c484d962f2856d32ad1fab0564349ccdfe70dbd6 WatchSource:0}: Error finding container 0b2cdb471bafb75ab5c977e7c484d962f2856d32ad1fab0564349ccdfe70dbd6: Status 404 returned error can't find the container with id 0b2cdb471bafb75ab5c977e7c484d962f2856d32ad1fab0564349ccdfe70dbd6
Jul 24 17:53:35.856485 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:35.856385    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cc64c6669-ghcws" event={"ID":"d8913974-7669-449d-8129-b56ff58193e2","Type":"ContainerStarted","Data":"d96e0a56500c7300267b1e147304388fff272514e9c7be0f62349fbece0b1ae4"}
Jul 24 17:53:35.856485 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:35.856442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cc64c6669-ghcws" event={"ID":"d8913974-7669-449d-8129-b56ff58193e2","Type":"ContainerStarted","Data":"0b2cdb471bafb75ab5c977e7c484d962f2856d32ad1fab0564349ccdfe70dbd6"}
Jul 24 17:53:35.876146 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:35.876087    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cc64c6669-ghcws" podStartSLOduration=1.301029797 podStartE2EDuration="1.876064589s" totalImagesPullingTime="575.034792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:53:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:53:35.873641348 +0000 UTC m=+638.396999838" watchObservedRunningTime="2026-07-24 17:53:35.876064589 +0000 UTC m=+638.399423068"
Jul 24 17:53:36.470061 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:36.469973    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:53:36.562727 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:36.562691    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:53:37.626963 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:37.626921    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:53:37.725430 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:53:37.725389    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:24.621941 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:24.621900    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bf5764cb5-lz4wz"]
Jul 24 17:54:24.622735 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:24.622162    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-bf5764cb5-lz4wz" podUID="55c5941b-f87c-4d4c-83d9-c400ea3d482e" containerName="manager" containerID="cri-o://5d873b49dd5d1b6090310d56e519d9748dd40c8855790fec508d83b595bf5c20" gracePeriod=30
Jul 24 17:54:24.870041 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:24.870018    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf5764cb5-lz4wz"
Jul 24 17:54:24.951207 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:24.951118    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55c5941b-f87c-4d4c-83d9-c400ea3d482e-webhook-cert\") pod \"55c5941b-f87c-4d4c-83d9-c400ea3d482e\" (UID: \"55c5941b-f87c-4d4c-83d9-c400ea3d482e\") "
Jul 24 17:54:24.951207 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:24.951192    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfthc\" (UniqueName: \"kubernetes.io/projected/55c5941b-f87c-4d4c-83d9-c400ea3d482e-kube-api-access-lfthc\") pod \"55c5941b-f87c-4d4c-83d9-c400ea3d482e\" (UID: \"55c5941b-f87c-4d4c-83d9-c400ea3d482e\") "
Jul 24 17:54:24.953091 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:24.953056    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c5941b-f87c-4d4c-83d9-c400ea3d482e-kube-api-access-lfthc" (OuterVolumeSpecName: "kube-api-access-lfthc") pod "55c5941b-f87c-4d4c-83d9-c400ea3d482e" (UID: "55c5941b-f87c-4d4c-83d9-c400ea3d482e"). InnerVolumeSpecName "kube-api-access-lfthc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:54:24.953223 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:24.953119    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c5941b-f87c-4d4c-83d9-c400ea3d482e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "55c5941b-f87c-4d4c-83d9-c400ea3d482e" (UID: "55c5941b-f87c-4d4c-83d9-c400ea3d482e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:54:25.052153 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:25.052121    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfthc\" (UniqueName: \"kubernetes.io/projected/55c5941b-f87c-4d4c-83d9-c400ea3d482e-kube-api-access-lfthc\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:54:25.052153 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:25.052147    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55c5941b-f87c-4d4c-83d9-c400ea3d482e-webhook-cert\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:54:25.100415 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:25.100381    2578 generic.go:356] "Generic (PLEG): container finished" podID="55c5941b-f87c-4d4c-83d9-c400ea3d482e" containerID="5d873b49dd5d1b6090310d56e519d9748dd40c8855790fec508d83b595bf5c20" exitCode=0
Jul 24 17:54:25.100569 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:25.100434    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf5764cb5-lz4wz"
Jul 24 17:54:25.100569 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:25.100474    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf5764cb5-lz4wz" event={"ID":"55c5941b-f87c-4d4c-83d9-c400ea3d482e","Type":"ContainerDied","Data":"5d873b49dd5d1b6090310d56e519d9748dd40c8855790fec508d83b595bf5c20"}
Jul 24 17:54:25.100569 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:25.100529    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf5764cb5-lz4wz" event={"ID":"55c5941b-f87c-4d4c-83d9-c400ea3d482e","Type":"ContainerDied","Data":"3dab2c280b51f843ff5b1fb411ae2af0bea8d38b5573934f873288af48201232"}
Jul 24 17:54:25.100569 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:25.100555    2578 scope.go:117] "RemoveContainer" containerID="5d873b49dd5d1b6090310d56e519d9748dd40c8855790fec508d83b595bf5c20"
Jul 24 17:54:25.110709 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:25.110687    2578 scope.go:117] "RemoveContainer" containerID="5d873b49dd5d1b6090310d56e519d9748dd40c8855790fec508d83b595bf5c20"
Jul 24 17:54:25.111003 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:54:25.110984    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d873b49dd5d1b6090310d56e519d9748dd40c8855790fec508d83b595bf5c20\": container with ID starting with 5d873b49dd5d1b6090310d56e519d9748dd40c8855790fec508d83b595bf5c20 not found: ID does not exist" containerID="5d873b49dd5d1b6090310d56e519d9748dd40c8855790fec508d83b595bf5c20"
Jul 24 17:54:25.111085 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:25.111015    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d873b49dd5d1b6090310d56e519d9748dd40c8855790fec508d83b595bf5c20"} err="failed to get container status \"5d873b49dd5d1b6090310d56e519d9748dd40c8855790fec508d83b595bf5c20\": rpc error: code = NotFound desc = could not find container \"5d873b49dd5d1b6090310d56e519d9748dd40c8855790fec508d83b595bf5c20\": container with ID starting with 5d873b49dd5d1b6090310d56e519d9748dd40c8855790fec508d83b595bf5c20 not found: ID does not exist"
Jul 24 17:54:25.124030 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:25.124004    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bf5764cb5-lz4wz"]
Jul 24 17:54:25.131208 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:25.131186    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bf5764cb5-lz4wz"]
Jul 24 17:54:25.967195 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:25.967163    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55c5941b-f87c-4d4c-83d9-c400ea3d482e" path="/var/lib/kubelet/pods/55c5941b-f87c-4d4c-83d9-c400ea3d482e/volumes"
Jul 24 17:54:57.849159 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:57.849072    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:57.950148 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:54:57.950117    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:07.726481 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:07.726420    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:07.793459 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:07.793436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:25.961758 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:25.961725    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bf5764cb5-vl6lx"]
Jul 24 17:55:25.962153 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:25.962126    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c5941b-f87c-4d4c-83d9-c400ea3d482e" containerName="manager"
Jul 24 17:55:25.962153 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:25.962138    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c5941b-f87c-4d4c-83d9-c400ea3d482e" containerName="manager"
Jul 24 17:55:25.962226 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:25.962208    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c5941b-f87c-4d4c-83d9-c400ea3d482e" containerName="manager"
Jul 24 17:55:25.965385 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:25.965359    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf5764cb5-vl6lx"
Jul 24 17:55:25.969040 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:25.969015    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 17:55:25.969173 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:25.969015    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 17:55:25.969173 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:25.969019    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8cxvz\""
Jul 24 17:55:25.975954 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:25.975931    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bf5764cb5-vl6lx"]
Jul 24 17:55:25.986806 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:25.986762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8566x\" (UniqueName: \"kubernetes.io/projected/dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd-kube-api-access-8566x\") pod \"maas-controller-bf5764cb5-vl6lx\" (UID: \"dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd\") " pod="opendatahub/maas-controller-bf5764cb5-vl6lx"
Jul 24 17:55:25.986971 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:25.986950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd-webhook-cert\") pod \"maas-controller-bf5764cb5-vl6lx\" (UID: \"dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd\") " pod="opendatahub/maas-controller-bf5764cb5-vl6lx"
Jul 24 17:55:26.087877 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:26.087834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8566x\" (UniqueName: \"kubernetes.io/projected/dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd-kube-api-access-8566x\") pod \"maas-controller-bf5764cb5-vl6lx\" (UID: \"dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd\") " pod="opendatahub/maas-controller-bf5764cb5-vl6lx"
Jul 24 17:55:26.088040 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:26.087908    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd-webhook-cert\") pod \"maas-controller-bf5764cb5-vl6lx\" (UID: \"dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd\") " pod="opendatahub/maas-controller-bf5764cb5-vl6lx"
Jul 24 17:55:26.090320 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:26.090300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd-webhook-cert\") pod \"maas-controller-bf5764cb5-vl6lx\" (UID: \"dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd\") " pod="opendatahub/maas-controller-bf5764cb5-vl6lx"
Jul 24 17:55:26.097891 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:26.097862    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8566x\" (UniqueName: \"kubernetes.io/projected/dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd-kube-api-access-8566x\") pod \"maas-controller-bf5764cb5-vl6lx\" (UID: \"dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd\") " pod="opendatahub/maas-controller-bf5764cb5-vl6lx"
Jul 24 17:55:26.277596 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:26.277512    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf5764cb5-vl6lx"
Jul 24 17:55:26.413417 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:26.413394    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bf5764cb5-vl6lx"]
Jul 24 17:55:26.415793 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:55:26.415751    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2dbeb1_b5b1_4fad_a29b_9f0c9da042cd.slice/crio-0b08750466a8951100d92891e526d0d15a26b8b173fe9120aedd87ab04cd93e0 WatchSource:0}: Error finding container 0b08750466a8951100d92891e526d0d15a26b8b173fe9120aedd87ab04cd93e0: Status 404 returned error can't find the container with id 0b08750466a8951100d92891e526d0d15a26b8b173fe9120aedd87ab04cd93e0
Jul 24 17:55:26.417098 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:26.417078    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 17:55:27.394012 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:27.393974    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf5764cb5-vl6lx" event={"ID":"dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd","Type":"ContainerStarted","Data":"d432c1bd5bbbe6cfdc515634f3921b9ccbe6dac32132c9f7ad64e78a93515dc0"}
Jul 24 17:55:27.394442 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:27.394026    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf5764cb5-vl6lx" event={"ID":"dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd","Type":"ContainerStarted","Data":"0b08750466a8951100d92891e526d0d15a26b8b173fe9120aedd87ab04cd93e0"}
Jul 24 17:55:27.394442 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:27.394085    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bf5764cb5-vl6lx"
Jul 24 17:55:27.414195 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:27.414147    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bf5764cb5-vl6lx" podStartSLOduration=1.962793505 podStartE2EDuration="2.414131535s" totalImagesPullingTime="451.33803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:55:27.413519318 +0000 UTC m=+749.936877797" watchObservedRunningTime="2026-07-24 17:55:27.414131535 +0000 UTC m=+749.937490053"
Jul 24 17:55:38.402182 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:38.402150    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bf5764cb5-vl6lx"
Jul 24 17:55:42.939828 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:42.939756    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:43.047056 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:43.047021    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bf5764cb5-vl6lx"]
Jul 24 17:55:43.047228 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:43.047197    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-bf5764cb5-vl6lx" podUID="dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd" containerName="manager" containerID="cri-o://d432c1bd5bbbe6cfdc515634f3921b9ccbe6dac32132c9f7ad64e78a93515dc0" gracePeriod=30
Jul 24 17:55:43.295295 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:43.295271    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf5764cb5-vl6lx"
Jul 24 17:55:43.442719 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:43.442687    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd-webhook-cert\") pod \"dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd\" (UID: \"dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd\") "
Jul 24 17:55:43.442889 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:43.442732    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8566x\" (UniqueName: \"kubernetes.io/projected/dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd-kube-api-access-8566x\") pod \"dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd\" (UID: \"dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd\") "
Jul 24 17:55:43.444817 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:43.444770    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd" (UID: "dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:55:43.444817 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:43.444804    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd-kube-api-access-8566x" (OuterVolumeSpecName: "kube-api-access-8566x") pod "dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd" (UID: "dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd"). InnerVolumeSpecName "kube-api-access-8566x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:55:43.483829 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:43.483758    2578 generic.go:356] "Generic (PLEG): container finished" podID="dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd" containerID="d432c1bd5bbbe6cfdc515634f3921b9ccbe6dac32132c9f7ad64e78a93515dc0" exitCode=0
Jul 24 17:55:43.483829 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:43.483809    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf5764cb5-vl6lx"
Jul 24 17:55:43.483829 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:43.483813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf5764cb5-vl6lx" event={"ID":"dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd","Type":"ContainerDied","Data":"d432c1bd5bbbe6cfdc515634f3921b9ccbe6dac32132c9f7ad64e78a93515dc0"}
Jul 24 17:55:43.484064 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:43.483865    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf5764cb5-vl6lx" event={"ID":"dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd","Type":"ContainerDied","Data":"0b08750466a8951100d92891e526d0d15a26b8b173fe9120aedd87ab04cd93e0"}
Jul 24 17:55:43.484064 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:43.483887    2578 scope.go:117] "RemoveContainer" containerID="d432c1bd5bbbe6cfdc515634f3921b9ccbe6dac32132c9f7ad64e78a93515dc0"
Jul 24 17:55:43.493054 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:43.493038    2578 scope.go:117] "RemoveContainer" containerID="d432c1bd5bbbe6cfdc515634f3921b9ccbe6dac32132c9f7ad64e78a93515dc0"
Jul 24 17:55:43.493299 ip-10-0-137-93 kubenswrapper[2578]: E0724 17:55:43.493276    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d432c1bd5bbbe6cfdc515634f3921b9ccbe6dac32132c9f7ad64e78a93515dc0\": container with ID starting with d432c1bd5bbbe6cfdc515634f3921b9ccbe6dac32132c9f7ad64e78a93515dc0 not found: ID does not exist" containerID="d432c1bd5bbbe6cfdc515634f3921b9ccbe6dac32132c9f7ad64e78a93515dc0"
Jul 24 17:55:43.493398 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:43.493306    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d432c1bd5bbbe6cfdc515634f3921b9ccbe6dac32132c9f7ad64e78a93515dc0"} err="failed to get container status \"d432c1bd5bbbe6cfdc515634f3921b9ccbe6dac32132c9f7ad64e78a93515dc0\": rpc error: code = NotFound desc = could not find container \"d432c1bd5bbbe6cfdc515634f3921b9ccbe6dac32132c9f7ad64e78a93515dc0\": container with ID starting with d432c1bd5bbbe6cfdc515634f3921b9ccbe6dac32132c9f7ad64e78a93515dc0 not found: ID does not exist"
Jul 24 17:55:43.505628 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:43.505603    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bf5764cb5-vl6lx"]
Jul 24 17:55:43.509468 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:43.509449    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bf5764cb5-vl6lx"]
Jul 24 17:55:43.543655 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:43.543630    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8566x\" (UniqueName: \"kubernetes.io/projected/dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd-kube-api-access-8566x\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:55:43.543655 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:43.543653    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd-webhook-cert\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 17:55:43.968403 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:43.968368    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd" path="/var/lib/kubelet/pods/dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd/volumes"
Jul 24 17:55:44.209567 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:44.209533    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:44.331155 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:44.331120    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bf5764cb5-v55w9"]
Jul 24 17:55:44.331535 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:44.331521    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd" containerName="manager"
Jul 24 17:55:44.331579 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:44.331536    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd" containerName="manager"
Jul 24 17:55:44.331614 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:44.331606    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc2dbeb1-b5b1-4fad-a29b-9f0c9da042cd" containerName="manager"
Jul 24 17:55:44.335951 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:44.335935    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf5764cb5-v55w9"
Jul 24 17:55:44.339872 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:44.339849    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 17:55:44.339872 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:44.339858    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8cxvz\""
Jul 24 17:55:44.340048 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:44.339859    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 17:55:44.344453 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:44.344433    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bf5764cb5-v55w9"]
Jul 24 17:55:44.451211 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:44.451176    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ada65a6-5dbf-472c-90e9-108a5e01948b-webhook-cert\") pod \"maas-controller-bf5764cb5-v55w9\" (UID: \"2ada65a6-5dbf-472c-90e9-108a5e01948b\") " pod="opendatahub/maas-controller-bf5764cb5-v55w9"
Jul 24 17:55:44.451368 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:44.451330    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csw9r\" (UniqueName: \"kubernetes.io/projected/2ada65a6-5dbf-472c-90e9-108a5e01948b-kube-api-access-csw9r\") pod \"maas-controller-bf5764cb5-v55w9\" (UID: \"2ada65a6-5dbf-472c-90e9-108a5e01948b\") " pod="opendatahub/maas-controller-bf5764cb5-v55w9"
Jul 24 17:55:44.552699 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:44.552668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csw9r\" (UniqueName: \"kubernetes.io/projected/2ada65a6-5dbf-472c-90e9-108a5e01948b-kube-api-access-csw9r\") pod \"maas-controller-bf5764cb5-v55w9\" (UID: \"2ada65a6-5dbf-472c-90e9-108a5e01948b\") " pod="opendatahub/maas-controller-bf5764cb5-v55w9"
Jul 24 17:55:44.552874 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:44.552711    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ada65a6-5dbf-472c-90e9-108a5e01948b-webhook-cert\") pod \"maas-controller-bf5764cb5-v55w9\" (UID: \"2ada65a6-5dbf-472c-90e9-108a5e01948b\") " pod="opendatahub/maas-controller-bf5764cb5-v55w9"
Jul 24 17:55:44.555232 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:44.555209    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ada65a6-5dbf-472c-90e9-108a5e01948b-webhook-cert\") pod \"maas-controller-bf5764cb5-v55w9\" (UID: \"2ada65a6-5dbf-472c-90e9-108a5e01948b\") " pod="opendatahub/maas-controller-bf5764cb5-v55w9"
Jul 24 17:55:44.561053 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:44.561028    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csw9r\" (UniqueName: \"kubernetes.io/projected/2ada65a6-5dbf-472c-90e9-108a5e01948b-kube-api-access-csw9r\") pod \"maas-controller-bf5764cb5-v55w9\" (UID: \"2ada65a6-5dbf-472c-90e9-108a5e01948b\") " pod="opendatahub/maas-controller-bf5764cb5-v55w9"
Jul 24 17:55:44.647773 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:44.647710    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf5764cb5-v55w9"
Jul 24 17:55:44.779823 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:44.779794    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bf5764cb5-v55w9"]
Jul 24 17:55:44.781724 ip-10-0-137-93 kubenswrapper[2578]: W0724 17:55:44.781690    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ada65a6_5dbf_472c_90e9_108a5e01948b.slice/crio-26ce314aa3764b5b737d7e1e80408a754cf1e31d2530523423f65a1ba06ad6cb WatchSource:0}: Error finding container 26ce314aa3764b5b737d7e1e80408a754cf1e31d2530523423f65a1ba06ad6cb: Status 404 returned error can't find the container with id 26ce314aa3764b5b737d7e1e80408a754cf1e31d2530523423f65a1ba06ad6cb
Jul 24 17:55:45.495875 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:45.495766    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf5764cb5-v55w9" event={"ID":"2ada65a6-5dbf-472c-90e9-108a5e01948b","Type":"ContainerStarted","Data":"88d1adfa3af489edbfd0543b80c0b40f128cd37c15fcb2f7244c119b9d638b47"}
Jul 24 17:55:45.496237 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:45.495934    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bf5764cb5-v55w9"
Jul 24 17:55:45.496237 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:45.495971    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf5764cb5-v55w9" event={"ID":"2ada65a6-5dbf-472c-90e9-108a5e01948b","Type":"ContainerStarted","Data":"26ce314aa3764b5b737d7e1e80408a754cf1e31d2530523423f65a1ba06ad6cb"}
Jul 24 17:55:45.516017 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:45.515968    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bf5764cb5-v55w9" podStartSLOduration=1.208639242 podStartE2EDuration="1.515954302s" totalImagesPullingTime="307.31506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:55:45.513384304 +0000 UTC m=+768.036742782" watchObservedRunningTime="2026-07-24 17:55:45.515954302 +0000 UTC m=+768.039312788"
Jul 24 17:55:56.502166 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:56.502134    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bf5764cb5-v55w9"
Jul 24 17:55:57.697556 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:57.697524    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:57.756679 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:55:57.756648    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:23.170257 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:56:23.170219    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:23.269994 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:56:23.269952    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:23.370334 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:56:23.370302    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:29.288494 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:56:29.288463    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:29.354039 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:56:29.354000    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:29.445319 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:56:29.445285    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:29.652421 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:56:29.652390    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:35.374235 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:56:35.374194    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:35.474411 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:56:35.474380    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 17:58:29.890889 ip-10-0-137-93 kubenswrapper[2578]: I0724 17:58:29.890857    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:00:08.869750 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:08.869712    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:00:08.953840 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:08.953805    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:00:21.660013 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:21.659977    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr"]
Jul 24 18:00:21.660471 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:21.660182    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr" podUID="6b05503a-70dd-4d52-84a2-97214db91af3" containerName="manager" containerID="cri-o://d4b8f40cdab48d96e66ea112f27edd8db7a5afcafad41ad63050ff5955eebfac" gracePeriod=10
Jul 24 18:00:23.219180 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:23.219154    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr"
Jul 24 18:00:23.309104 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:23.309072    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b05503a-70dd-4d52-84a2-97214db91af3-extensions-socket-volume\") pod \"6b05503a-70dd-4d52-84a2-97214db91af3\" (UID: \"6b05503a-70dd-4d52-84a2-97214db91af3\") "
Jul 24 18:00:23.309297 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:23.309136    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2n4c\" (UniqueName: \"kubernetes.io/projected/6b05503a-70dd-4d52-84a2-97214db91af3-kube-api-access-f2n4c\") pod \"6b05503a-70dd-4d52-84a2-97214db91af3\" (UID: \"6b05503a-70dd-4d52-84a2-97214db91af3\") "
Jul 24 18:00:23.309527 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:23.309501    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b05503a-70dd-4d52-84a2-97214db91af3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6b05503a-70dd-4d52-84a2-97214db91af3" (UID: "6b05503a-70dd-4d52-84a2-97214db91af3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:00:23.311255 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:23.311222    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b05503a-70dd-4d52-84a2-97214db91af3-kube-api-access-f2n4c" (OuterVolumeSpecName: "kube-api-access-f2n4c") pod "6b05503a-70dd-4d52-84a2-97214db91af3" (UID: "6b05503a-70dd-4d52-84a2-97214db91af3"). InnerVolumeSpecName "kube-api-access-f2n4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:00:23.410338 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:23.410302    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b05503a-70dd-4d52-84a2-97214db91af3-extensions-socket-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:00:23.410338 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:23.410331    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2n4c\" (UniqueName: \"kubernetes.io/projected/6b05503a-70dd-4d52-84a2-97214db91af3-kube-api-access-f2n4c\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:00:23.844099 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:23.844063    2578 generic.go:356] "Generic (PLEG): container finished" podID="6b05503a-70dd-4d52-84a2-97214db91af3" containerID="d4b8f40cdab48d96e66ea112f27edd8db7a5afcafad41ad63050ff5955eebfac" exitCode=0
Jul 24 18:00:23.844099 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:23.844100    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr" event={"ID":"6b05503a-70dd-4d52-84a2-97214db91af3","Type":"ContainerDied","Data":"d4b8f40cdab48d96e66ea112f27edd8db7a5afcafad41ad63050ff5955eebfac"}
Jul 24 18:00:23.844363 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:23.844112    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr"
Jul 24 18:00:23.844363 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:23.844138    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr" event={"ID":"6b05503a-70dd-4d52-84a2-97214db91af3","Type":"ContainerDied","Data":"cfec97cb0d5cd197b48e2f2ca3121db024a7f3640290515ec44929094546b8c8"}
Jul 24 18:00:23.844363 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:23.844158    2578 scope.go:117] "RemoveContainer" containerID="d4b8f40cdab48d96e66ea112f27edd8db7a5afcafad41ad63050ff5955eebfac"
Jul 24 18:00:23.854511 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:23.854490    2578 scope.go:117] "RemoveContainer" containerID="d4b8f40cdab48d96e66ea112f27edd8db7a5afcafad41ad63050ff5955eebfac"
Jul 24 18:00:23.854774 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:00:23.854754    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4b8f40cdab48d96e66ea112f27edd8db7a5afcafad41ad63050ff5955eebfac\": container with ID starting with d4b8f40cdab48d96e66ea112f27edd8db7a5afcafad41ad63050ff5955eebfac not found: ID does not exist" containerID="d4b8f40cdab48d96e66ea112f27edd8db7a5afcafad41ad63050ff5955eebfac"
Jul 24 18:00:23.854865 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:23.854810    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4b8f40cdab48d96e66ea112f27edd8db7a5afcafad41ad63050ff5955eebfac"} err="failed to get container status \"d4b8f40cdab48d96e66ea112f27edd8db7a5afcafad41ad63050ff5955eebfac\": rpc error: code = NotFound desc = could not find container \"d4b8f40cdab48d96e66ea112f27edd8db7a5afcafad41ad63050ff5955eebfac\": container with ID starting with d4b8f40cdab48d96e66ea112f27edd8db7a5afcafad41ad63050ff5955eebfac not found: ID does not exist"
Jul 24 18:00:23.865956 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:23.865933    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr"]
Jul 24 18:00:23.872741 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:23.872717    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-lwtxr"]
Jul 24 18:00:23.967000 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:00:23.966969    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b05503a-70dd-4d52-84a2-97214db91af3" path="/var/lib/kubelet/pods/6b05503a-70dd-4d52-84a2-97214db91af3/volumes"
Jul 24 18:01:22.822150 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:22.822115    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4zwpl"]
Jul 24 18:01:22.822541 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:22.822505    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b05503a-70dd-4d52-84a2-97214db91af3" containerName="manager"
Jul 24 18:01:22.822541 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:22.822516    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b05503a-70dd-4d52-84a2-97214db91af3" containerName="manager"
Jul 24 18:01:22.822620 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:22.822577    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b05503a-70dd-4d52-84a2-97214db91af3" containerName="manager"
Jul 24 18:01:22.827849 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:22.827821    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4zwpl"
Jul 24 18:01:22.830399 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:22.830371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ff82\" (UniqueName: \"kubernetes.io/projected/a7fbfd17-a32c-4639-b06c-d2b9dfe14d3a-kube-api-access-9ff82\") pod \"kuadrant-operator-controller-manager-68cf589589-4zwpl\" (UID: \"a7fbfd17-a32c-4639-b06c-d2b9dfe14d3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4zwpl"
Jul 24 18:01:22.830545 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:22.830417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7fbfd17-a32c-4639-b06c-d2b9dfe14d3a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-4zwpl\" (UID: \"a7fbfd17-a32c-4639-b06c-d2b9dfe14d3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4zwpl"
Jul 24 18:01:22.830849 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:22.830830    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4znsw\""
Jul 24 18:01:22.838902 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:22.838874    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4zwpl"]
Jul 24 18:01:22.931572 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:22.931536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ff82\" (UniqueName: \"kubernetes.io/projected/a7fbfd17-a32c-4639-b06c-d2b9dfe14d3a-kube-api-access-9ff82\") pod \"kuadrant-operator-controller-manager-68cf589589-4zwpl\" (UID: \"a7fbfd17-a32c-4639-b06c-d2b9dfe14d3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4zwpl"
Jul 24 18:01:22.931572 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:22.931588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7fbfd17-a32c-4639-b06c-d2b9dfe14d3a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-4zwpl\" (UID: \"a7fbfd17-a32c-4639-b06c-d2b9dfe14d3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4zwpl"
Jul 24 18:01:22.932019 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:22.931999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7fbfd17-a32c-4639-b06c-d2b9dfe14d3a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-4zwpl\" (UID: \"a7fbfd17-a32c-4639-b06c-d2b9dfe14d3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4zwpl"
Jul 24 18:01:22.940698 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:22.940675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ff82\" (UniqueName: \"kubernetes.io/projected/a7fbfd17-a32c-4639-b06c-d2b9dfe14d3a-kube-api-access-9ff82\") pod \"kuadrant-operator-controller-manager-68cf589589-4zwpl\" (UID: \"a7fbfd17-a32c-4639-b06c-d2b9dfe14d3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4zwpl"
Jul 24 18:01:23.140733 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:23.140649    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4zwpl"
Jul 24 18:01:23.311430 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:01:23.311397    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7fbfd17_a32c_4639_b06c_d2b9dfe14d3a.slice/crio-726b64031563907706b5c628992d00e01f25dec270dabf3f97c2bb032272a577 WatchSource:0}: Error finding container 726b64031563907706b5c628992d00e01f25dec270dabf3f97c2bb032272a577: Status 404 returned error can't find the container with id 726b64031563907706b5c628992d00e01f25dec270dabf3f97c2bb032272a577
Jul 24 18:01:23.314452 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:23.314418    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4zwpl"]
Jul 24 18:01:24.141239 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:24.141205    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4zwpl" event={"ID":"a7fbfd17-a32c-4639-b06c-d2b9dfe14d3a","Type":"ContainerStarted","Data":"4126e43aba3def597972132417209682df2dc92abbec14fc136f075aa238c8ba"}
Jul 24 18:01:24.141648 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:24.141264    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4zwpl" event={"ID":"a7fbfd17-a32c-4639-b06c-d2b9dfe14d3a","Type":"ContainerStarted","Data":"726b64031563907706b5c628992d00e01f25dec270dabf3f97c2bb032272a577"}
Jul 24 18:01:24.141648 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:24.141370    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4zwpl"
Jul 24 18:01:24.166117 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:24.166069    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4zwpl" podStartSLOduration=2.166054036 podStartE2EDuration="2.166054036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:01:24.163179986 +0000 UTC m=+1106.686538467" watchObservedRunningTime="2026-07-24 18:01:24.166054036 +0000 UTC m=+1106.689412513"
Jul 24 18:01:35.148867 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:01:35.148833    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4zwpl"
Jul 24 18:02:17.184143 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:02:17.184106    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:04:02.802044 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:04:02.801954    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:04:06.781387 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:04:06.781340    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:04:30.472193 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:04:30.472155    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:04:31.422471 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:04:31.422436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:05:27.797655 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:05:27.797617    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:35.334248 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:07:35.334207    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:51.811285 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:07:51.811248    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:51.868316 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:07:51.868279    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:56.010676 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:07:56.010640    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:59.261015 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:07:59.260974    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:59.365888 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:07:59.365849    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:59.736146 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:07:59.736116    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:05.466698 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:05.466663    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:05.536125 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:05.536092    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:05.713598 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:05.713568    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:06.050429 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:06.050397    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:06.160473 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:06.160439    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:08.477259 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:08.477231    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:08.591763 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:08.591729    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:08.726226 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:08.726186    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"]
Jul 24 18:08:08.730186 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:08.730141    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"
Jul 24 18:08:08.739799 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:08.739763    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-096012db-serving-cert\""
Jul 24 18:08:08.749593 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:08.749569    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"]
Jul 24 18:08:08.770254 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:08.770218    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:08.813981 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:08.813950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f951a4f9-b495-448e-957f-3e86af3c487f-maas-api-tls\") pod \"maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj\" (UID: \"f951a4f9-b495-448e-957f-3e86af3c487f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"
Jul 24 18:08:08.813981 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:08.813986    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f951a4f9-b495-448e-957f-3e86af3c487f-openshift-service-ca\") pod \"maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj\" (UID: \"f951a4f9-b495-448e-957f-3e86af3c487f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"
Jul 24 18:08:08.814198 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:08.814014    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7g42\" (UniqueName: \"kubernetes.io/projected/f951a4f9-b495-448e-957f-3e86af3c487f-kube-api-access-g7g42\") pod \"maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj\" (UID: \"f951a4f9-b495-448e-957f-3e86af3c487f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"
Jul 24 18:08:08.914512 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:08.914475    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f951a4f9-b495-448e-957f-3e86af3c487f-openshift-service-ca\") pod \"maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj\" (UID: \"f951a4f9-b495-448e-957f-3e86af3c487f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"
Jul 24 18:08:08.914713 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:08.914525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7g42\" (UniqueName: \"kubernetes.io/projected/f951a4f9-b495-448e-957f-3e86af3c487f-kube-api-access-g7g42\") pod \"maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj\" (UID: \"f951a4f9-b495-448e-957f-3e86af3c487f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"
Jul 24 18:08:08.914713 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:08.914632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f951a4f9-b495-448e-957f-3e86af3c487f-maas-api-tls\") pod \"maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj\" (UID: \"f951a4f9-b495-448e-957f-3e86af3c487f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"
Jul 24 18:08:08.915260 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:08.915235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f951a4f9-b495-448e-957f-3e86af3c487f-openshift-service-ca\") pod \"maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj\" (UID: \"f951a4f9-b495-448e-957f-3e86af3c487f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"
Jul 24 18:08:08.917211 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:08.917181    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f951a4f9-b495-448e-957f-3e86af3c487f-maas-api-tls\") pod \"maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj\" (UID: \"f951a4f9-b495-448e-957f-3e86af3c487f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"
Jul 24 18:08:08.929590 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:08.929556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:08.929887 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:08.929870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7g42\" (UniqueName: \"kubernetes.io/projected/f951a4f9-b495-448e-957f-3e86af3c487f-kube-api-access-g7g42\") pod \"maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj\" (UID: \"f951a4f9-b495-448e-957f-3e86af3c487f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"
Jul 24 18:08:09.040774 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.040675    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"
Jul 24 18:08:09.178636 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.178609    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"]
Jul 24 18:08:09.180727 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:08:09.180699    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf951a4f9_b495_448e_957f_3e86af3c487f.slice/crio-cc9d8459a9300ebe5b40e4049d52767e0ea632fbc2e2d623a976cf0935da1368 WatchSource:0}: Error finding container cc9d8459a9300ebe5b40e4049d52767e0ea632fbc2e2d623a976cf0935da1368: Status 404 returned error can't find the container with id cc9d8459a9300ebe5b40e4049d52767e0ea632fbc2e2d623a976cf0935da1368
Jul 24 18:08:09.182055 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.182035    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 18:08:09.466668 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.466625    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:09.585876 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.585837    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:09.596718 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.596692    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr"]
Jul 24 18:08:09.601735 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.601717    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr"
Jul 24 18:08:09.604420 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.604398    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-096012db\""
Jul 24 18:08:09.614366 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.614285    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9"]
Jul 24 18:08:09.619988 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.619955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8549ea-e02c-4f45-828a-d0c6e244c612-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr\" (UID: \"bd8549ea-e02c-4f45-828a-d0c6e244c612\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr"
Jul 24 18:08:09.620107 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.620033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p9tw\" (UniqueName: \"kubernetes.io/projected/bd8549ea-e02c-4f45-828a-d0c6e244c612-kube-api-access-4p9tw\") pod \"payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr\" (UID: \"bd8549ea-e02c-4f45-828a-d0c6e244c612\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr"
Jul 24 18:08:09.620830 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.620804    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9"
Jul 24 18:08:09.622372 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.622350    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr"]
Jul 24 18:08:09.624172 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.624152    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-096012db-dockercfg-rmq95\""
Jul 24 18:08:09.636286 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.636071    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9"]
Jul 24 18:08:09.721069 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.721035    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc9bd7b-c57a-4f1a-8da4-911831cf0934-plugins-config-volume\") pod \"payload-processing-e2e-ait-096012db-77bc58697b-rjbd9\" (UID: \"ecc9bd7b-c57a-4f1a-8da4-911831cf0934\") " pod="openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9"
Jul 24 18:08:09.721207 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.721144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvmpj\" (UniqueName: \"kubernetes.io/projected/ecc9bd7b-c57a-4f1a-8da4-911831cf0934-kube-api-access-tvmpj\") pod \"payload-processing-e2e-ait-096012db-77bc58697b-rjbd9\" (UID: \"ecc9bd7b-c57a-4f1a-8da4-911831cf0934\") " pod="openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9"
Jul 24 18:08:09.721207 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.721202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8549ea-e02c-4f45-828a-d0c6e244c612-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr\" (UID: \"bd8549ea-e02c-4f45-828a-d0c6e244c612\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr"
Jul 24 18:08:09.721321 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.721227    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p9tw\" (UniqueName: \"kubernetes.io/projected/bd8549ea-e02c-4f45-828a-d0c6e244c612-kube-api-access-4p9tw\") pod \"payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr\" (UID: \"bd8549ea-e02c-4f45-828a-d0c6e244c612\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr"
Jul 24 18:08:09.722122 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.722091    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8549ea-e02c-4f45-828a-d0c6e244c612-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr\" (UID: \"bd8549ea-e02c-4f45-828a-d0c6e244c612\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr"
Jul 24 18:08:09.730291 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.730269    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p9tw\" (UniqueName: \"kubernetes.io/projected/bd8549ea-e02c-4f45-828a-d0c6e244c612-kube-api-access-4p9tw\") pod \"payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr\" (UID: \"bd8549ea-e02c-4f45-828a-d0c6e244c612\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr"
Jul 24 18:08:09.822282 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.822242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc9bd7b-c57a-4f1a-8da4-911831cf0934-plugins-config-volume\") pod \"payload-processing-e2e-ait-096012db-77bc58697b-rjbd9\" (UID: \"ecc9bd7b-c57a-4f1a-8da4-911831cf0934\") " pod="openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9"
Jul 24 18:08:09.822479 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.822308    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvmpj\" (UniqueName: \"kubernetes.io/projected/ecc9bd7b-c57a-4f1a-8da4-911831cf0934-kube-api-access-tvmpj\") pod \"payload-processing-e2e-ait-096012db-77bc58697b-rjbd9\" (UID: \"ecc9bd7b-c57a-4f1a-8da4-911831cf0934\") " pod="openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9"
Jul 24 18:08:09.823070 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.823041    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc9bd7b-c57a-4f1a-8da4-911831cf0934-plugins-config-volume\") pod \"payload-processing-e2e-ait-096012db-77bc58697b-rjbd9\" (UID: \"ecc9bd7b-c57a-4f1a-8da4-911831cf0934\") " pod="openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9"
Jul 24 18:08:09.831222 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.831198    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvmpj\" (UniqueName: \"kubernetes.io/projected/ecc9bd7b-c57a-4f1a-8da4-911831cf0934-kube-api-access-tvmpj\") pod \"payload-processing-e2e-ait-096012db-77bc58697b-rjbd9\" (UID: \"ecc9bd7b-c57a-4f1a-8da4-911831cf0934\") " pod="openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9"
Jul 24 18:08:09.914521 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.914487    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr"
Jul 24 18:08:09.931976 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:09.931947    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9"
Jul 24 18:08:10.082096 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:10.082021    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr"]
Jul 24 18:08:10.085033 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:08:10.085001    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8549ea_e02c_4f45_828a_d0c6e244c612.slice/crio-f5bae59c2cdeef8dc59e23031863ecc970c5e52552665bc6364074f4249f08fc WatchSource:0}: Error finding container f5bae59c2cdeef8dc59e23031863ecc970c5e52552665bc6364074f4249f08fc: Status 404 returned error can't find the container with id f5bae59c2cdeef8dc59e23031863ecc970c5e52552665bc6364074f4249f08fc
Jul 24 18:08:10.103408 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:10.103285    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9"]
Jul 24 18:08:10.107338 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:08:10.107314    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc9bd7b_c57a_4f1a_8da4_911831cf0934.slice/crio-196f87c89575789128e3fb59414d5c1911dfa1cfa03d640fc1f25e346327f312 WatchSource:0}: Error finding container 196f87c89575789128e3fb59414d5c1911dfa1cfa03d640fc1f25e346327f312: Status 404 returned error can't find the container with id 196f87c89575789128e3fb59414d5c1911dfa1cfa03d640fc1f25e346327f312
Jul 24 18:08:10.152882 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:10.152841    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr" event={"ID":"bd8549ea-e02c-4f45-828a-d0c6e244c612","Type":"ContainerStarted","Data":"f5bae59c2cdeef8dc59e23031863ecc970c5e52552665bc6364074f4249f08fc"}
Jul 24 18:08:10.154323 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:10.154294    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9" event={"ID":"ecc9bd7b-c57a-4f1a-8da4-911831cf0934","Type":"ContainerStarted","Data":"196f87c89575789128e3fb59414d5c1911dfa1cfa03d640fc1f25e346327f312"}
Jul 24 18:08:10.156573 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:10.156550    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj" event={"ID":"f951a4f9-b495-448e-957f-3e86af3c487f","Type":"ContainerStarted","Data":"2dc989d2a21355b02a257001c55d529f65345da99e0eb00e52cafadfbee7220f"}
Jul 24 18:08:10.156675 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:10.156590    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj" event={"ID":"f951a4f9-b495-448e-957f-3e86af3c487f","Type":"ContainerStarted","Data":"cc9d8459a9300ebe5b40e4049d52767e0ea632fbc2e2d623a976cf0935da1368"}
Jul 24 18:08:10.156821 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:10.156697    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"
Jul 24 18:08:10.178460 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:10.178416    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj" podStartSLOduration=1.670411799 podStartE2EDuration="2.178397319s" totalImagesPullingTime="507.98552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:08:10.176718965 +0000 UTC m=+1512.700077445" watchObservedRunningTime="2026-07-24 18:08:10.178397319 +0000 UTC m=+1512.701755798"
Jul 24 18:08:11.163484 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:11.163437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9" event={"ID":"ecc9bd7b-c57a-4f1a-8da4-911831cf0934","Type":"ContainerStarted","Data":"7ed11ff72878d819c43fc65e91c2ad470e0a19bb7861c143278c01144494b6f4"}
Jul 24 18:08:11.163894 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:11.163519    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9"
Jul 24 18:08:11.165045 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:11.165020    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr" event={"ID":"bd8549ea-e02c-4f45-828a-d0c6e244c612","Type":"ContainerStarted","Data":"6eb3d3c3fc4bc648a8ba83dd504bfea387fff979137fd9cf6150e7a8e423334b"}
Jul 24 18:08:11.165232 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:11.165209    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr"
Jul 24 18:08:11.180997 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:11.180951    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9" podStartSLOduration=2.180936142 podStartE2EDuration="2.180936142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:08:11.180088715 +0000 UTC m=+1513.703447194" watchObservedRunningTime="2026-07-24 18:08:11.180936142 +0000 UTC m=+1513.704294620"
Jul 24 18:08:11.196554 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:11.196507    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr" podStartSLOduration=2.19649129 podStartE2EDuration="2.19649129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:08:11.195412067 +0000 UTC m=+1513.718770545" watchObservedRunningTime="2026-07-24 18:08:11.19649129 +0000 UTC m=+1513.719849785"
Jul 24 18:08:16.170707 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:16.170674    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"
Jul 24 18:08:22.171689 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:22.171604    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9"
Jul 24 18:08:22.172109 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:22.171798    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr"
Jul 24 18:08:44.676164 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:44.676129    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"]
Jul 24 18:08:44.676665 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:44.676394    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj" podUID="f951a4f9-b495-448e-957f-3e86af3c487f" containerName="maas-api" containerID="cri-o://2dc989d2a21355b02a257001c55d529f65345da99e0eb00e52cafadfbee7220f" gracePeriod=30
Jul 24 18:08:44.709992 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:44.709962    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:44.735440 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:44.735402    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9"]
Jul 24 18:08:44.735686 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:44.735654    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9" podUID="ecc9bd7b-c57a-4f1a-8da4-911831cf0934" containerName="payload-processing" containerID="cri-o://7ed11ff72878d819c43fc65e91c2ad470e0a19bb7861c143278c01144494b6f4" gracePeriod=30
Jul 24 18:08:44.742650 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:44.742614    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr"]
Jul 24 18:08:44.742960 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:44.742897    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr" podUID="bd8549ea-e02c-4f45-828a-d0c6e244c612" containerName="payload-pre-processing" containerID="cri-o://6eb3d3c3fc4bc648a8ba83dd504bfea387fff979137fd9cf6150e7a8e423334b" gracePeriod=30
Jul 24 18:08:44.783957 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:44.783927    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:44.843416 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:44.843382    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:45.005135 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.005104    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"
Jul 24 18:08:45.053365 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.053338    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9"
Jul 24 18:08:45.062604 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.062581    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr"
Jul 24 18:08:45.143870 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.143837    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7g42\" (UniqueName: \"kubernetes.io/projected/f951a4f9-b495-448e-957f-3e86af3c487f-kube-api-access-g7g42\") pod \"f951a4f9-b495-448e-957f-3e86af3c487f\" (UID: \"f951a4f9-b495-448e-957f-3e86af3c487f\") "
Jul 24 18:08:45.144054 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.143877    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8549ea-e02c-4f45-828a-d0c6e244c612-plugins-config-volume\") pod \"bd8549ea-e02c-4f45-828a-d0c6e244c612\" (UID: \"bd8549ea-e02c-4f45-828a-d0c6e244c612\") "
Jul 24 18:08:45.144054 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.143909    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvmpj\" (UniqueName: \"kubernetes.io/projected/ecc9bd7b-c57a-4f1a-8da4-911831cf0934-kube-api-access-tvmpj\") pod \"ecc9bd7b-c57a-4f1a-8da4-911831cf0934\" (UID: \"ecc9bd7b-c57a-4f1a-8da4-911831cf0934\") "
Jul 24 18:08:45.144054 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.143936    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc9bd7b-c57a-4f1a-8da4-911831cf0934-plugins-config-volume\") pod \"ecc9bd7b-c57a-4f1a-8da4-911831cf0934\" (UID: \"ecc9bd7b-c57a-4f1a-8da4-911831cf0934\") "
Jul 24 18:08:45.144054 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.143990    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f951a4f9-b495-448e-957f-3e86af3c487f-maas-api-tls\") pod \"f951a4f9-b495-448e-957f-3e86af3c487f\" (UID: \"f951a4f9-b495-448e-957f-3e86af3c487f\") "
Jul 24 18:08:45.144054 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.144014    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f951a4f9-b495-448e-957f-3e86af3c487f-openshift-service-ca\") pod \"f951a4f9-b495-448e-957f-3e86af3c487f\" (UID: \"f951a4f9-b495-448e-957f-3e86af3c487f\") "
Jul 24 18:08:45.144054 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.144028    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4p9tw\" (UniqueName: \"kubernetes.io/projected/bd8549ea-e02c-4f45-828a-d0c6e244c612-kube-api-access-4p9tw\") pod \"bd8549ea-e02c-4f45-828a-d0c6e244c612\" (UID: \"bd8549ea-e02c-4f45-828a-d0c6e244c612\") "
Jul 24 18:08:45.144381 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.144357    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8549ea-e02c-4f45-828a-d0c6e244c612-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bd8549ea-e02c-4f45-828a-d0c6e244c612" (UID: "bd8549ea-e02c-4f45-828a-d0c6e244c612"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:08:45.144451 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.144414    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f951a4f9-b495-448e-957f-3e86af3c487f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f951a4f9-b495-448e-957f-3e86af3c487f" (UID: "f951a4f9-b495-448e-957f-3e86af3c487f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:08:45.144451 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.144421    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc9bd7b-c57a-4f1a-8da4-911831cf0934-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ecc9bd7b-c57a-4f1a-8da4-911831cf0934" (UID: "ecc9bd7b-c57a-4f1a-8da4-911831cf0934"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:08:45.146205 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.146178    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8549ea-e02c-4f45-828a-d0c6e244c612-kube-api-access-4p9tw" (OuterVolumeSpecName: "kube-api-access-4p9tw") pod "bd8549ea-e02c-4f45-828a-d0c6e244c612" (UID: "bd8549ea-e02c-4f45-828a-d0c6e244c612"). InnerVolumeSpecName "kube-api-access-4p9tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:08:45.146314 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.146247    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f951a4f9-b495-448e-957f-3e86af3c487f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f951a4f9-b495-448e-957f-3e86af3c487f" (UID: "f951a4f9-b495-448e-957f-3e86af3c487f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:08:45.146495 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.146467    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f951a4f9-b495-448e-957f-3e86af3c487f-kube-api-access-g7g42" (OuterVolumeSpecName: "kube-api-access-g7g42") pod "f951a4f9-b495-448e-957f-3e86af3c487f" (UID: "f951a4f9-b495-448e-957f-3e86af3c487f"). InnerVolumeSpecName "kube-api-access-g7g42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:08:45.146602 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.146474    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc9bd7b-c57a-4f1a-8da4-911831cf0934-kube-api-access-tvmpj" (OuterVolumeSpecName: "kube-api-access-tvmpj") pod "ecc9bd7b-c57a-4f1a-8da4-911831cf0934" (UID: "ecc9bd7b-c57a-4f1a-8da4-911831cf0934"). InnerVolumeSpecName "kube-api-access-tvmpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:08:45.245347 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.245261    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc9bd7b-c57a-4f1a-8da4-911831cf0934-plugins-config-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:08:45.245347 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.245293    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f951a4f9-b495-448e-957f-3e86af3c487f-maas-api-tls\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:08:45.245347 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.245305    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f951a4f9-b495-448e-957f-3e86af3c487f-openshift-service-ca\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:08:45.245347 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.245314    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p9tw\" (UniqueName: \"kubernetes.io/projected/bd8549ea-e02c-4f45-828a-d0c6e244c612-kube-api-access-4p9tw\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:08:45.245347 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.245324    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7g42\" (UniqueName: \"kubernetes.io/projected/f951a4f9-b495-448e-957f-3e86af3c487f-kube-api-access-g7g42\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:08:45.245347 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.245333    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8549ea-e02c-4f45-828a-d0c6e244c612-plugins-config-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:08:45.245347 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.245341    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvmpj\" (UniqueName: \"kubernetes.io/projected/ecc9bd7b-c57a-4f1a-8da4-911831cf0934-kube-api-access-tvmpj\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:08:45.342222 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.342184    2578 generic.go:356] "Generic (PLEG): container finished" podID="bd8549ea-e02c-4f45-828a-d0c6e244c612" containerID="6eb3d3c3fc4bc648a8ba83dd504bfea387fff979137fd9cf6150e7a8e423334b" exitCode=0
Jul 24 18:08:45.342416 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.342228    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr"
Jul 24 18:08:45.342416 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.342273    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr" event={"ID":"bd8549ea-e02c-4f45-828a-d0c6e244c612","Type":"ContainerDied","Data":"6eb3d3c3fc4bc648a8ba83dd504bfea387fff979137fd9cf6150e7a8e423334b"}
Jul 24 18:08:45.342416 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.342334    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr" event={"ID":"bd8549ea-e02c-4f45-828a-d0c6e244c612","Type":"ContainerDied","Data":"f5bae59c2cdeef8dc59e23031863ecc970c5e52552665bc6364074f4249f08fc"}
Jul 24 18:08:45.342416 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.342364    2578 scope.go:117] "RemoveContainer" containerID="6eb3d3c3fc4bc648a8ba83dd504bfea387fff979137fd9cf6150e7a8e423334b"
Jul 24 18:08:45.343770 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.343744    2578 generic.go:356] "Generic (PLEG): container finished" podID="ecc9bd7b-c57a-4f1a-8da4-911831cf0934" containerID="7ed11ff72878d819c43fc65e91c2ad470e0a19bb7861c143278c01144494b6f4" exitCode=0
Jul 24 18:08:45.343907 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.343809    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9"
Jul 24 18:08:45.343907 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.343816    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9" event={"ID":"ecc9bd7b-c57a-4f1a-8da4-911831cf0934","Type":"ContainerDied","Data":"7ed11ff72878d819c43fc65e91c2ad470e0a19bb7861c143278c01144494b6f4"}
Jul 24 18:08:45.343907 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.343863    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9" event={"ID":"ecc9bd7b-c57a-4f1a-8da4-911831cf0934","Type":"ContainerDied","Data":"196f87c89575789128e3fb59414d5c1911dfa1cfa03d640fc1f25e346327f312"}
Jul 24 18:08:45.345363 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.345336    2578 generic.go:356] "Generic (PLEG): container finished" podID="f951a4f9-b495-448e-957f-3e86af3c487f" containerID="2dc989d2a21355b02a257001c55d529f65345da99e0eb00e52cafadfbee7220f" exitCode=0
Jul 24 18:08:45.345467 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.345371    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj" event={"ID":"f951a4f9-b495-448e-957f-3e86af3c487f","Type":"ContainerDied","Data":"2dc989d2a21355b02a257001c55d529f65345da99e0eb00e52cafadfbee7220f"}
Jul 24 18:08:45.345467 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.345378    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"
Jul 24 18:08:45.345467 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.345407    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj" event={"ID":"f951a4f9-b495-448e-957f-3e86af3c487f","Type":"ContainerDied","Data":"cc9d8459a9300ebe5b40e4049d52767e0ea632fbc2e2d623a976cf0935da1368"}
Jul 24 18:08:45.355972 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.355956    2578 scope.go:117] "RemoveContainer" containerID="6eb3d3c3fc4bc648a8ba83dd504bfea387fff979137fd9cf6150e7a8e423334b"
Jul 24 18:08:45.356278 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:08:45.356261    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb3d3c3fc4bc648a8ba83dd504bfea387fff979137fd9cf6150e7a8e423334b\": container with ID starting with 6eb3d3c3fc4bc648a8ba83dd504bfea387fff979137fd9cf6150e7a8e423334b not found: ID does not exist" containerID="6eb3d3c3fc4bc648a8ba83dd504bfea387fff979137fd9cf6150e7a8e423334b"
Jul 24 18:08:45.356350 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.356288    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb3d3c3fc4bc648a8ba83dd504bfea387fff979137fd9cf6150e7a8e423334b"} err="failed to get container status \"6eb3d3c3fc4bc648a8ba83dd504bfea387fff979137fd9cf6150e7a8e423334b\": rpc error: code = NotFound desc = could not find container \"6eb3d3c3fc4bc648a8ba83dd504bfea387fff979137fd9cf6150e7a8e423334b\": container with ID starting with 6eb3d3c3fc4bc648a8ba83dd504bfea387fff979137fd9cf6150e7a8e423334b not found: ID does not exist"
Jul 24 18:08:45.356350 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.356309    2578 scope.go:117] "RemoveContainer" containerID="7ed11ff72878d819c43fc65e91c2ad470e0a19bb7861c143278c01144494b6f4"
Jul 24 18:08:45.364222 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.364206    2578 scope.go:117] "RemoveContainer" containerID="7ed11ff72878d819c43fc65e91c2ad470e0a19bb7861c143278c01144494b6f4"
Jul 24 18:08:45.364476 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:08:45.364457    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ed11ff72878d819c43fc65e91c2ad470e0a19bb7861c143278c01144494b6f4\": container with ID starting with 7ed11ff72878d819c43fc65e91c2ad470e0a19bb7861c143278c01144494b6f4 not found: ID does not exist" containerID="7ed11ff72878d819c43fc65e91c2ad470e0a19bb7861c143278c01144494b6f4"
Jul 24 18:08:45.364525 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.364483    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed11ff72878d819c43fc65e91c2ad470e0a19bb7861c143278c01144494b6f4"} err="failed to get container status \"7ed11ff72878d819c43fc65e91c2ad470e0a19bb7861c143278c01144494b6f4\": rpc error: code = NotFound desc = could not find container \"7ed11ff72878d819c43fc65e91c2ad470e0a19bb7861c143278c01144494b6f4\": container with ID starting with 7ed11ff72878d819c43fc65e91c2ad470e0a19bb7861c143278c01144494b6f4 not found: ID does not exist"
Jul 24 18:08:45.364525 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.364499    2578 scope.go:117] "RemoveContainer" containerID="2dc989d2a21355b02a257001c55d529f65345da99e0eb00e52cafadfbee7220f"
Jul 24 18:08:45.372331 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.372313    2578 scope.go:117] "RemoveContainer" containerID="2dc989d2a21355b02a257001c55d529f65345da99e0eb00e52cafadfbee7220f"
Jul 24 18:08:45.372588 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:08:45.372568    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc989d2a21355b02a257001c55d529f65345da99e0eb00e52cafadfbee7220f\": container with ID starting with 2dc989d2a21355b02a257001c55d529f65345da99e0eb00e52cafadfbee7220f not found: ID does not exist" containerID="2dc989d2a21355b02a257001c55d529f65345da99e0eb00e52cafadfbee7220f"
Jul 24 18:08:45.372654 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.372598    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc989d2a21355b02a257001c55d529f65345da99e0eb00e52cafadfbee7220f"} err="failed to get container status \"2dc989d2a21355b02a257001c55d529f65345da99e0eb00e52cafadfbee7220f\": rpc error: code = NotFound desc = could not find container \"2dc989d2a21355b02a257001c55d529f65345da99e0eb00e52cafadfbee7220f\": container with ID starting with 2dc989d2a21355b02a257001c55d529f65345da99e0eb00e52cafadfbee7220f not found: ID does not exist"
Jul 24 18:08:45.374333 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.374312    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr"]
Jul 24 18:08:45.377917 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.377897    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-096012db-657f5469dd-bfbcr"]
Jul 24 18:08:45.390375 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.390348    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9"]
Jul 24 18:08:45.394103 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.394082    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-096012db-77bc58697b-rjbd9"]
Jul 24 18:08:45.405544 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.405522    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"]
Jul 24 18:08:45.409638 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.409615    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-096012db-6b9c78bcbd-kd8jj"]
Jul 24 18:08:45.967895 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.967819    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd8549ea-e02c-4f45-828a-d0c6e244c612" path="/var/lib/kubelet/pods/bd8549ea-e02c-4f45-828a-d0c6e244c612/volumes"
Jul 24 18:08:45.968251 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.968192    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecc9bd7b-c57a-4f1a-8da4-911831cf0934" path="/var/lib/kubelet/pods/ecc9bd7b-c57a-4f1a-8da4-911831cf0934/volumes"
Jul 24 18:08:45.968532 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:45.968519    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f951a4f9-b495-448e-957f-3e86af3c487f" path="/var/lib/kubelet/pods/f951a4f9-b495-448e-957f-3e86af3c487f/volumes"
Jul 24 18:08:49.223207 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:49.223172    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:49.376474 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:49.376446    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:49.627808 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:49.627748    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:49.900740 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:49.900634    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:49.994956 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:49.994910    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:50.120227 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:50.120188    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:50.472237 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:50.472198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:50.974395 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:50.974363    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:52.845459 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:52.845415    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:52.921559 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:52.921523    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"]
Jul 24 18:08:52.922019 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:52.922002    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecc9bd7b-c57a-4f1a-8da4-911831cf0934" containerName="payload-processing"
Jul 24 18:08:52.922073 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:52.922021    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecc9bd7b-c57a-4f1a-8da4-911831cf0934" containerName="payload-processing"
Jul 24 18:08:52.922073 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:52.922038    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f951a4f9-b495-448e-957f-3e86af3c487f" containerName="maas-api"
Jul 24 18:08:52.922073 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:52.922044    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f951a4f9-b495-448e-957f-3e86af3c487f" containerName="maas-api"
Jul 24 18:08:52.922073 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:52.922053    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd8549ea-e02c-4f45-828a-d0c6e244c612" containerName="payload-pre-processing"
Jul 24 18:08:52.922073 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:52.922059    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd8549ea-e02c-4f45-828a-d0c6e244c612" containerName="payload-pre-processing"
Jul 24 18:08:52.922229 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:52.922123    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f951a4f9-b495-448e-957f-3e86af3c487f" containerName="maas-api"
Jul 24 18:08:52.922229 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:52.922134    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecc9bd7b-c57a-4f1a-8da4-911831cf0934" containerName="payload-processing"
Jul 24 18:08:52.922229 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:52.922143    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd8549ea-e02c-4f45-828a-d0c6e244c612" containerName="payload-pre-processing"
Jul 24 18:08:52.926755 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:52.926732    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"
Jul 24 18:08:52.929426 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:52.929404    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-35d23e85-serving-cert\""
Jul 24 18:08:52.937571 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:52.937521    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"]
Jul 24 18:08:52.969224 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:52.969192    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:53.121501 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.121397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpz97\" (UniqueName: \"kubernetes.io/projected/ac3262f5-e192-4e63-92bb-c8f3e3750e92-kube-api-access-gpz97\") pod \"maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s\" (UID: \"ac3262f5-e192-4e63-92bb-c8f3e3750e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"
Jul 24 18:08:53.121669 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.121532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3262f5-e192-4e63-92bb-c8f3e3750e92-openshift-service-ca\") pod \"maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s\" (UID: \"ac3262f5-e192-4e63-92bb-c8f3e3750e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"
Jul 24 18:08:53.121669 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.121565    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3262f5-e192-4e63-92bb-c8f3e3750e92-maas-api-tls\") pod \"maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s\" (UID: \"ac3262f5-e192-4e63-92bb-c8f3e3750e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"
Jul 24 18:08:53.170257 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.170228    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:53.222593 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.222550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpz97\" (UniqueName: \"kubernetes.io/projected/ac3262f5-e192-4e63-92bb-c8f3e3750e92-kube-api-access-gpz97\") pod \"maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s\" (UID: \"ac3262f5-e192-4e63-92bb-c8f3e3750e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"
Jul 24 18:08:53.222770 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.222636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3262f5-e192-4e63-92bb-c8f3e3750e92-openshift-service-ca\") pod \"maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s\" (UID: \"ac3262f5-e192-4e63-92bb-c8f3e3750e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"
Jul 24 18:08:53.222770 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.222656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3262f5-e192-4e63-92bb-c8f3e3750e92-maas-api-tls\") pod \"maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s\" (UID: \"ac3262f5-e192-4e63-92bb-c8f3e3750e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"
Jul 24 18:08:53.223301 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.223279    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3262f5-e192-4e63-92bb-c8f3e3750e92-openshift-service-ca\") pod \"maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s\" (UID: \"ac3262f5-e192-4e63-92bb-c8f3e3750e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"
Jul 24 18:08:53.225194 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.225172    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3262f5-e192-4e63-92bb-c8f3e3750e92-maas-api-tls\") pod \"maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s\" (UID: \"ac3262f5-e192-4e63-92bb-c8f3e3750e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"
Jul 24 18:08:53.237010 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.236981    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpz97\" (UniqueName: \"kubernetes.io/projected/ac3262f5-e192-4e63-92bb-c8f3e3750e92-kube-api-access-gpz97\") pod \"maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s\" (UID: \"ac3262f5-e192-4e63-92bb-c8f3e3750e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"
Jul 24 18:08:53.238670 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.238644    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"
Jul 24 18:08:53.396966 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.396940    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"]
Jul 24 18:08:53.398930 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:08:53.398870    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3262f5_e192_4e63_92bb_c8f3e3750e92.slice/crio-bd577516834face63f25d257ca9fd580c4ef9384205337c1ae3cd2b97a700ff5 WatchSource:0}: Error finding container bd577516834face63f25d257ca9fd580c4ef9384205337c1ae3cd2b97a700ff5: Status 404 returned error can't find the container with id bd577516834face63f25d257ca9fd580c4ef9384205337c1ae3cd2b97a700ff5
Jul 24 18:08:53.559138 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.559100    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:53.656112 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.656015    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:53.678940 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.678900    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn"]
Jul 24 18:08:53.683610 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.683586    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn"
Jul 24 18:08:53.687185 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.686980    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-35d23e85\""
Jul 24 18:08:53.687185 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.686980    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-35d23e85-dockercfg-4kxt9\""
Jul 24 18:08:53.696333 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.696302    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn"]
Jul 24 18:08:53.698485 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.698456    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw"]
Jul 24 18:08:53.702237 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.702214    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw"
Jul 24 18:08:53.714288 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.714258    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw"]
Jul 24 18:08:53.828180 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.828137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkm8s\" (UniqueName: \"kubernetes.io/projected/0f60a4eb-084e-40a8-98b6-665dce45fe0d-kube-api-access-kkm8s\") pod \"payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw\" (UID: \"0f60a4eb-084e-40a8-98b6-665dce45fe0d\") " pod="openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw"
Jul 24 18:08:53.828379 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.828222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f60a4eb-084e-40a8-98b6-665dce45fe0d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw\" (UID: \"0f60a4eb-084e-40a8-98b6-665dce45fe0d\") " pod="openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw"
Jul 24 18:08:53.828379 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.828242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5da7628-4354-4d55-a67d-fe5033751b8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn\" (UID: \"b5da7628-4354-4d55-a67d-fe5033751b8f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn"
Jul 24 18:08:53.828379 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.828329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxjx4\" (UniqueName: \"kubernetes.io/projected/b5da7628-4354-4d55-a67d-fe5033751b8f-kube-api-access-zxjx4\") pod \"payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn\" (UID: \"b5da7628-4354-4d55-a67d-fe5033751b8f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn"
Jul 24 18:08:53.929618 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.929530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f60a4eb-084e-40a8-98b6-665dce45fe0d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw\" (UID: \"0f60a4eb-084e-40a8-98b6-665dce45fe0d\") " pod="openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw"
Jul 24 18:08:53.929618 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.929566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5da7628-4354-4d55-a67d-fe5033751b8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn\" (UID: \"b5da7628-4354-4d55-a67d-fe5033751b8f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn"
Jul 24 18:08:53.930183 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.929622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxjx4\" (UniqueName: \"kubernetes.io/projected/b5da7628-4354-4d55-a67d-fe5033751b8f-kube-api-access-zxjx4\") pod \"payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn\" (UID: \"b5da7628-4354-4d55-a67d-fe5033751b8f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn"
Jul 24 18:08:53.930183 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.929651    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkm8s\" (UniqueName: \"kubernetes.io/projected/0f60a4eb-084e-40a8-98b6-665dce45fe0d-kube-api-access-kkm8s\") pod \"payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw\" (UID: \"0f60a4eb-084e-40a8-98b6-665dce45fe0d\") " pod="openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw"
Jul 24 18:08:53.930375 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.930304    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f60a4eb-084e-40a8-98b6-665dce45fe0d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw\" (UID: \"0f60a4eb-084e-40a8-98b6-665dce45fe0d\") " pod="openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw"
Jul 24 18:08:53.930512 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.930489    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5da7628-4354-4d55-a67d-fe5033751b8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn\" (UID: \"b5da7628-4354-4d55-a67d-fe5033751b8f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn"
Jul 24 18:08:53.938059 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.938037    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxjx4\" (UniqueName: \"kubernetes.io/projected/b5da7628-4354-4d55-a67d-fe5033751b8f-kube-api-access-zxjx4\") pod \"payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn\" (UID: \"b5da7628-4354-4d55-a67d-fe5033751b8f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn"
Jul 24 18:08:53.938156 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.938059    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkm8s\" (UniqueName: \"kubernetes.io/projected/0f60a4eb-084e-40a8-98b6-665dce45fe0d-kube-api-access-kkm8s\") pod \"payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw\" (UID: \"0f60a4eb-084e-40a8-98b6-665dce45fe0d\") " pod="openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw"
Jul 24 18:08:53.995517 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:53.995476    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn"
Jul 24 18:08:54.014664 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:54.014626    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw"
Jul 24 18:08:54.156677 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:54.156639    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn"]
Jul 24 18:08:54.158630 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:08:54.158596    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5da7628_4354_4d55_a67d_fe5033751b8f.slice/crio-0eb76365925fa87a2d0dba39641ad68d571994ef895f95b4ee91305d10a2bf7a WatchSource:0}: Error finding container 0eb76365925fa87a2d0dba39641ad68d571994ef895f95b4ee91305d10a2bf7a: Status 404 returned error can't find the container with id 0eb76365925fa87a2d0dba39641ad68d571994ef895f95b4ee91305d10a2bf7a
Jul 24 18:08:54.181026 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:54.180994    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw"]
Jul 24 18:08:54.190884 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:08:54.190853    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f60a4eb_084e_40a8_98b6_665dce45fe0d.slice/crio-9c11e9efab9fd4379bfe65c6bd4b90bf698e08d6c30bd2ae419d5b7b1f3baa8b WatchSource:0}: Error finding container 9c11e9efab9fd4379bfe65c6bd4b90bf698e08d6c30bd2ae419d5b7b1f3baa8b: Status 404 returned error can't find the container with id 9c11e9efab9fd4379bfe65c6bd4b90bf698e08d6c30bd2ae419d5b7b1f3baa8b
Jul 24 18:08:54.394478 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:54.394049    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw" event={"ID":"0f60a4eb-084e-40a8-98b6-665dce45fe0d","Type":"ContainerStarted","Data":"55ffd406c0ce5afb2e653b9b4a5122fb24c12b5c6464a68a114357aa6ed39300"}
Jul 24 18:08:54.394478 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:54.394113    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw" event={"ID":"0f60a4eb-084e-40a8-98b6-665dce45fe0d","Type":"ContainerStarted","Data":"9c11e9efab9fd4379bfe65c6bd4b90bf698e08d6c30bd2ae419d5b7b1f3baa8b"}
Jul 24 18:08:54.394829 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:54.394702    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw"
Jul 24 18:08:54.396282 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:54.396218    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn" event={"ID":"b5da7628-4354-4d55-a67d-fe5033751b8f","Type":"ContainerStarted","Data":"8c71d3dade28c97858f0f8d08d5f50f9767d2cba7c4b5bf7ff63003c081125d5"}
Jul 24 18:08:54.396282 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:54.396274    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn" event={"ID":"b5da7628-4354-4d55-a67d-fe5033751b8f","Type":"ContainerStarted","Data":"0eb76365925fa87a2d0dba39641ad68d571994ef895f95b4ee91305d10a2bf7a"}
Jul 24 18:08:54.396469 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:54.396305    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn"
Jul 24 18:08:54.398554 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:54.398474    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s" event={"ID":"ac3262f5-e192-4e63-92bb-c8f3e3750e92","Type":"ContainerStarted","Data":"d3081777459ac10ec20250ab87748cd81970e11a40e4ffd0ea2fbe9a4c9373c0"}
Jul 24 18:08:54.398554 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:54.398519    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s" event={"ID":"ac3262f5-e192-4e63-92bb-c8f3e3750e92","Type":"ContainerStarted","Data":"bd577516834face63f25d257ca9fd580c4ef9384205337c1ae3cd2b97a700ff5"}
Jul 24 18:08:54.398823 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:54.398766    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"
Jul 24 18:08:54.426266 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:54.426215    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw" podStartSLOduration=1.426199268 podStartE2EDuration="1.426199268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:08:54.425624598 +0000 UTC m=+1556.948983077" watchObservedRunningTime="2026-07-24 18:08:54.426199268 +0000 UTC m=+1556.949557747"
Jul 24 18:08:54.472344 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:08:54.472236    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn" podStartSLOduration=1.472220393 podStartE2EDuration="1.472220393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:08:54.469966238 +0000 UTC m=+1556.993324715" watchObservedRunningTime="2026-07-24 18:08:54.472220393 +0000 UTC m=+1556.995578871"
Jul 24 18:09:00.411589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:00.411551    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"
Jul 24 18:09:00.431489 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:00.431437    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s" podStartSLOduration=7.997251499 podStartE2EDuration="8.431421981s" totalImagesPullingTime="434.170482ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:08:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:08:54.509670943 +0000 UTC m=+1557.033029420" watchObservedRunningTime="2026-07-24 18:09:00.431421981 +0000 UTC m=+1562.954780459"
Jul 24 18:09:05.406919 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:05.406887    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn"
Jul 24 18:09:06.413643 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:06.413606    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw"
Jul 24 18:09:13.183117 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.183075    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"]
Jul 24 18:09:13.183584 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.183344    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s" podUID="ac3262f5-e192-4e63-92bb-c8f3e3750e92" containerName="maas-api" containerID="cri-o://d3081777459ac10ec20250ab87748cd81970e11a40e4ffd0ea2fbe9a4c9373c0" gracePeriod=30
Jul 24 18:09:13.268827 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.268798    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:13.337165 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.337127    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw"]
Jul 24 18:09:13.337427 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.337396    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw" podUID="0f60a4eb-084e-40a8-98b6-665dce45fe0d" containerName="payload-processing" containerID="cri-o://55ffd406c0ce5afb2e653b9b4a5122fb24c12b5c6464a68a114357aa6ed39300" gracePeriod=30
Jul 24 18:09:13.347177 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.347139    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn"]
Jul 24 18:09:13.347391 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.347363    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn" podUID="b5da7628-4354-4d55-a67d-fe5033751b8f" containerName="payload-pre-processing" containerID="cri-o://8c71d3dade28c97858f0f8d08d5f50f9767d2cba7c4b5bf7ff63003c081125d5" gracePeriod=30
Jul 24 18:09:13.479627 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.479598    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"
Jul 24 18:09:13.505120 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.504634    2578 generic.go:356] "Generic (PLEG): container finished" podID="ac3262f5-e192-4e63-92bb-c8f3e3750e92" containerID="d3081777459ac10ec20250ab87748cd81970e11a40e4ffd0ea2fbe9a4c9373c0" exitCode=0
Jul 24 18:09:13.505120 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.504798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s" event={"ID":"ac3262f5-e192-4e63-92bb-c8f3e3750e92","Type":"ContainerDied","Data":"d3081777459ac10ec20250ab87748cd81970e11a40e4ffd0ea2fbe9a4c9373c0"}
Jul 24 18:09:13.505120 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.504852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s" event={"ID":"ac3262f5-e192-4e63-92bb-c8f3e3750e92","Type":"ContainerDied","Data":"bd577516834face63f25d257ca9fd580c4ef9384205337c1ae3cd2b97a700ff5"}
Jul 24 18:09:13.505120 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.504887    2578 scope.go:117] "RemoveContainer" containerID="d3081777459ac10ec20250ab87748cd81970e11a40e4ffd0ea2fbe9a4c9373c0"
Jul 24 18:09:13.505120 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.505054    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"
Jul 24 18:09:13.514494 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.508417    2578 generic.go:356] "Generic (PLEG): container finished" podID="0f60a4eb-084e-40a8-98b6-665dce45fe0d" containerID="55ffd406c0ce5afb2e653b9b4a5122fb24c12b5c6464a68a114357aa6ed39300" exitCode=0
Jul 24 18:09:13.514494 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.508501    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw" event={"ID":"0f60a4eb-084e-40a8-98b6-665dce45fe0d","Type":"ContainerDied","Data":"55ffd406c0ce5afb2e653b9b4a5122fb24c12b5c6464a68a114357aa6ed39300"}
Jul 24 18:09:13.514494 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.511819    2578 generic.go:356] "Generic (PLEG): container finished" podID="b5da7628-4354-4d55-a67d-fe5033751b8f" containerID="8c71d3dade28c97858f0f8d08d5f50f9767d2cba7c4b5bf7ff63003c081125d5" exitCode=0
Jul 24 18:09:13.514494 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.511860    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn" event={"ID":"b5da7628-4354-4d55-a67d-fe5033751b8f","Type":"ContainerDied","Data":"8c71d3dade28c97858f0f8d08d5f50f9767d2cba7c4b5bf7ff63003c081125d5"}
Jul 24 18:09:13.522200 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.522167    2578 scope.go:117] "RemoveContainer" containerID="d3081777459ac10ec20250ab87748cd81970e11a40e4ffd0ea2fbe9a4c9373c0"
Jul 24 18:09:13.522599 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:09:13.522570    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3081777459ac10ec20250ab87748cd81970e11a40e4ffd0ea2fbe9a4c9373c0\": container with ID starting with d3081777459ac10ec20250ab87748cd81970e11a40e4ffd0ea2fbe9a4c9373c0 not found: ID does not exist" containerID="d3081777459ac10ec20250ab87748cd81970e11a40e4ffd0ea2fbe9a4c9373c0"
Jul 24 18:09:13.522661 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.522613    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3081777459ac10ec20250ab87748cd81970e11a40e4ffd0ea2fbe9a4c9373c0"} err="failed to get container status \"d3081777459ac10ec20250ab87748cd81970e11a40e4ffd0ea2fbe9a4c9373c0\": rpc error: code = NotFound desc = could not find container \"d3081777459ac10ec20250ab87748cd81970e11a40e4ffd0ea2fbe9a4c9373c0\": container with ID starting with d3081777459ac10ec20250ab87748cd81970e11a40e4ffd0ea2fbe9a4c9373c0 not found: ID does not exist"
Jul 24 18:09:13.626103 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.626077    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpz97\" (UniqueName: \"kubernetes.io/projected/ac3262f5-e192-4e63-92bb-c8f3e3750e92-kube-api-access-gpz97\") pod \"ac3262f5-e192-4e63-92bb-c8f3e3750e92\" (UID: \"ac3262f5-e192-4e63-92bb-c8f3e3750e92\") "
Jul 24 18:09:13.626235 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.626174    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3262f5-e192-4e63-92bb-c8f3e3750e92-openshift-service-ca\") pod \"ac3262f5-e192-4e63-92bb-c8f3e3750e92\" (UID: \"ac3262f5-e192-4e63-92bb-c8f3e3750e92\") "
Jul 24 18:09:13.626299 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.626237    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3262f5-e192-4e63-92bb-c8f3e3750e92-maas-api-tls\") pod \"ac3262f5-e192-4e63-92bb-c8f3e3750e92\" (UID: \"ac3262f5-e192-4e63-92bb-c8f3e3750e92\") "
Jul 24 18:09:13.626952 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.626829    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3262f5-e192-4e63-92bb-c8f3e3750e92-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ac3262f5-e192-4e63-92bb-c8f3e3750e92" (UID: "ac3262f5-e192-4e63-92bb-c8f3e3750e92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:09:13.628360 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.628334    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3262f5-e192-4e63-92bb-c8f3e3750e92-kube-api-access-gpz97" (OuterVolumeSpecName: "kube-api-access-gpz97") pod "ac3262f5-e192-4e63-92bb-c8f3e3750e92" (UID: "ac3262f5-e192-4e63-92bb-c8f3e3750e92"). InnerVolumeSpecName "kube-api-access-gpz97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:09:13.628534 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.628516    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac3262f5-e192-4e63-92bb-c8f3e3750e92-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ac3262f5-e192-4e63-92bb-c8f3e3750e92" (UID: "ac3262f5-e192-4e63-92bb-c8f3e3750e92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:09:13.640186 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.639573    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw"
Jul 24 18:09:13.642897 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.642878    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn"
Jul 24 18:09:13.727768 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.727674    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f60a4eb-084e-40a8-98b6-665dce45fe0d-plugins-config-volume\") pod \"0f60a4eb-084e-40a8-98b6-665dce45fe0d\" (UID: \"0f60a4eb-084e-40a8-98b6-665dce45fe0d\") "
Jul 24 18:09:13.727768 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.727727    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5da7628-4354-4d55-a67d-fe5033751b8f-plugins-config-volume\") pod \"b5da7628-4354-4d55-a67d-fe5033751b8f\" (UID: \"b5da7628-4354-4d55-a67d-fe5033751b8f\") "
Jul 24 18:09:13.728034 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.727771    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxjx4\" (UniqueName: \"kubernetes.io/projected/b5da7628-4354-4d55-a67d-fe5033751b8f-kube-api-access-zxjx4\") pod \"b5da7628-4354-4d55-a67d-fe5033751b8f\" (UID: \"b5da7628-4354-4d55-a67d-fe5033751b8f\") "
Jul 24 18:09:13.728034 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.727902    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkm8s\" (UniqueName: \"kubernetes.io/projected/0f60a4eb-084e-40a8-98b6-665dce45fe0d-kube-api-access-kkm8s\") pod \"0f60a4eb-084e-40a8-98b6-665dce45fe0d\" (UID: \"0f60a4eb-084e-40a8-98b6-665dce45fe0d\") "
Jul 24 18:09:13.728189 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.728164    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5da7628-4354-4d55-a67d-fe5033751b8f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b5da7628-4354-4d55-a67d-fe5033751b8f" (UID: "b5da7628-4354-4d55-a67d-fe5033751b8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:09:13.728238 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.728169    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f60a4eb-084e-40a8-98b6-665dce45fe0d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0f60a4eb-084e-40a8-98b6-665dce45fe0d" (UID: "0f60a4eb-084e-40a8-98b6-665dce45fe0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:09:13.728302 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.728287    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5da7628-4354-4d55-a67d-fe5033751b8f-plugins-config-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:09:13.728336 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.728307    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpz97\" (UniqueName: \"kubernetes.io/projected/ac3262f5-e192-4e63-92bb-c8f3e3750e92-kube-api-access-gpz97\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:09:13.728336 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.728316    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3262f5-e192-4e63-92bb-c8f3e3750e92-openshift-service-ca\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:09:13.728336 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.728326    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3262f5-e192-4e63-92bb-c8f3e3750e92-maas-api-tls\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:09:13.728336 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.728334    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f60a4eb-084e-40a8-98b6-665dce45fe0d-plugins-config-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:09:13.729809 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.729794    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5da7628-4354-4d55-a67d-fe5033751b8f-kube-api-access-zxjx4" (OuterVolumeSpecName: "kube-api-access-zxjx4") pod "b5da7628-4354-4d55-a67d-fe5033751b8f" (UID: "b5da7628-4354-4d55-a67d-fe5033751b8f"). InnerVolumeSpecName "kube-api-access-zxjx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:09:13.729865 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.729820    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f60a4eb-084e-40a8-98b6-665dce45fe0d-kube-api-access-kkm8s" (OuterVolumeSpecName: "kube-api-access-kkm8s") pod "0f60a4eb-084e-40a8-98b6-665dce45fe0d" (UID: "0f60a4eb-084e-40a8-98b6-665dce45fe0d"). InnerVolumeSpecName "kube-api-access-kkm8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:09:13.830215 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.830174    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxjx4\" (UniqueName: \"kubernetes.io/projected/b5da7628-4354-4d55-a67d-fe5033751b8f-kube-api-access-zxjx4\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:09:13.830215 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.830212    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkm8s\" (UniqueName: \"kubernetes.io/projected/0f60a4eb-084e-40a8-98b6-665dce45fe0d-kube-api-access-kkm8s\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:09:13.837278 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.837211    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"]
Jul 24 18:09:13.838226 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.838197    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-35d23e85-7fdf49b9bb-kkr4s"]
Jul 24 18:09:13.968111 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:13.968070    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac3262f5-e192-4e63-92bb-c8f3e3750e92" path="/var/lib/kubelet/pods/ac3262f5-e192-4e63-92bb-c8f3e3750e92/volumes"
Jul 24 18:09:14.518083 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:14.518053    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw"
Jul 24 18:09:14.518083 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:14.518069    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw" event={"ID":"0f60a4eb-084e-40a8-98b6-665dce45fe0d","Type":"ContainerDied","Data":"9c11e9efab9fd4379bfe65c6bd4b90bf698e08d6c30bd2ae419d5b7b1f3baa8b"}
Jul 24 18:09:14.518609 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:14.518138    2578 scope.go:117] "RemoveContainer" containerID="55ffd406c0ce5afb2e653b9b4a5122fb24c12b5c6464a68a114357aa6ed39300"
Jul 24 18:09:14.519653 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:14.519611    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn"
Jul 24 18:09:14.519653 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:14.519618    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn" event={"ID":"b5da7628-4354-4d55-a67d-fe5033751b8f","Type":"ContainerDied","Data":"0eb76365925fa87a2d0dba39641ad68d571994ef895f95b4ee91305d10a2bf7a"}
Jul 24 18:09:14.527272 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:14.527255    2578 scope.go:117] "RemoveContainer" containerID="8c71d3dade28c97858f0f8d08d5f50f9767d2cba7c4b5bf7ff63003c081125d5"
Jul 24 18:09:14.533675 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:14.533658    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:14.541424 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:14.541403    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn"]
Jul 24 18:09:14.545813 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:14.545793    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-35d23e85-69f55c746d-kkmfn"]
Jul 24 18:09:14.559041 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:14.559020    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw"]
Jul 24 18:09:14.561947 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:14.561926    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-35d23e85-7b9d664477-qnznw"]
Jul 24 18:09:15.968921 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:15.968882    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f60a4eb-084e-40a8-98b6-665dce45fe0d" path="/var/lib/kubelet/pods/0f60a4eb-084e-40a8-98b6-665dce45fe0d/volumes"
Jul 24 18:09:15.969574 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:15.969553    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5da7628-4354-4d55-a67d-fe5033751b8f" path="/var/lib/kubelet/pods/b5da7628-4354-4d55-a67d-fe5033751b8f/volumes"
Jul 24 18:09:17.728353 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:17.728323    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:17.889142 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:17.889113    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:18.099069 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:18.099028    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:18.352718 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:18.352626    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:18.663269 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:18.663191    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:18.768622 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:18.768590    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:21.188477 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.188447    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:21.285879 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.285851    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:21.358897 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.358858    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"]
Jul 24 18:09:21.359496 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.359475    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5da7628-4354-4d55-a67d-fe5033751b8f" containerName="payload-pre-processing"
Jul 24 18:09:21.359557 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.359500    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5da7628-4354-4d55-a67d-fe5033751b8f" containerName="payload-pre-processing"
Jul 24 18:09:21.359557 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.359543    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac3262f5-e192-4e63-92bb-c8f3e3750e92" containerName="maas-api"
Jul 24 18:09:21.359557 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.359552    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac3262f5-e192-4e63-92bb-c8f3e3750e92" containerName="maas-api"
Jul 24 18:09:21.359656 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.359568    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f60a4eb-084e-40a8-98b6-665dce45fe0d" containerName="payload-processing"
Jul 24 18:09:21.359656 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.359576    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f60a4eb-084e-40a8-98b6-665dce45fe0d" containerName="payload-processing"
Jul 24 18:09:21.359730 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.359669    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac3262f5-e192-4e63-92bb-c8f3e3750e92" containerName="maas-api"
Jul 24 18:09:21.359730 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.359682    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f60a4eb-084e-40a8-98b6-665dce45fe0d" containerName="payload-processing"
Jul 24 18:09:21.359730 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.359692    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5da7628-4354-4d55-a67d-fe5033751b8f" containerName="payload-pre-processing"
Jul 24 18:09:21.364603 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.364581    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"
Jul 24 18:09:21.367202 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.367172    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f37130a8-serving-cert\""
Jul 24 18:09:21.375340 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.375317    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"]
Jul 24 18:09:21.424214 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.424177    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:21.501926 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.501824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ce080c-b649-4050-92c8-067a0a826ad6-openshift-service-ca\") pod \"maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw\" (UID: \"90ce080c-b649-4050-92c8-067a0a826ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"
Jul 24 18:09:21.502106 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.502036    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttg8n\" (UniqueName: \"kubernetes.io/projected/90ce080c-b649-4050-92c8-067a0a826ad6-kube-api-access-ttg8n\") pod \"maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw\" (UID: \"90ce080c-b649-4050-92c8-067a0a826ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"
Jul 24 18:09:21.502106 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.502082    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ce080c-b649-4050-92c8-067a0a826ad6-maas-api-tls\") pod \"maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw\" (UID: \"90ce080c-b649-4050-92c8-067a0a826ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"
Jul 24 18:09:21.602984 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.602946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttg8n\" (UniqueName: \"kubernetes.io/projected/90ce080c-b649-4050-92c8-067a0a826ad6-kube-api-access-ttg8n\") pod \"maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw\" (UID: \"90ce080c-b649-4050-92c8-067a0a826ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"
Jul 24 18:09:21.602984 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.602983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ce080c-b649-4050-92c8-067a0a826ad6-maas-api-tls\") pod \"maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw\" (UID: \"90ce080c-b649-4050-92c8-067a0a826ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"
Jul 24 18:09:21.603202 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.603019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ce080c-b649-4050-92c8-067a0a826ad6-openshift-service-ca\") pod \"maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw\" (UID: \"90ce080c-b649-4050-92c8-067a0a826ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"
Jul 24 18:09:21.603202 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:09:21.603111    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-serving-cert: secret "maas-api-e2e-ait-f37130a8-serving-cert" not found
Jul 24 18:09:21.603202 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:09:21.603186    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90ce080c-b649-4050-92c8-067a0a826ad6-maas-api-tls podName:90ce080c-b649-4050-92c8-067a0a826ad6 nodeName:}" failed. No retries permitted until 2026-07-24 18:09:22.103166182 +0000 UTC m=+1584.626524639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90ce080c-b649-4050-92c8-067a0a826ad6-maas-api-tls") pod "maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw" (UID: "90ce080c-b649-4050-92c8-067a0a826ad6") : secret "maas-api-e2e-ait-f37130a8-serving-cert" not found
Jul 24 18:09:21.603829 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.603806    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ce080c-b649-4050-92c8-067a0a826ad6-openshift-service-ca\") pod \"maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw\" (UID: \"90ce080c-b649-4050-92c8-067a0a826ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"
Jul 24 18:09:21.614006 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.613984    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttg8n\" (UniqueName: \"kubernetes.io/projected/90ce080c-b649-4050-92c8-067a0a826ad6-kube-api-access-ttg8n\") pod \"maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw\" (UID: \"90ce080c-b649-4050-92c8-067a0a826ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"
Jul 24 18:09:21.893199 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:21.893140    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:22.024211 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.024162    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:22.108102 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.108062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ce080c-b649-4050-92c8-067a0a826ad6-maas-api-tls\") pod \"maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw\" (UID: \"90ce080c-b649-4050-92c8-067a0a826ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"
Jul 24 18:09:22.110493 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.110467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ce080c-b649-4050-92c8-067a0a826ad6-maas-api-tls\") pod \"maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw\" (UID: \"90ce080c-b649-4050-92c8-067a0a826ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"
Jul 24 18:09:22.254065 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.253962    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:22.274112 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.274081    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692"]
Jul 24 18:09:22.275988 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.275967    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"
Jul 24 18:09:22.278214 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.278198    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692"
Jul 24 18:09:22.280879 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.280860    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f37130a8-dockercfg-pkghr\""
Jul 24 18:09:22.281186 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.281166    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f37130a8\""
Jul 24 18:09:22.296459 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.296421    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692"]
Jul 24 18:09:22.410599 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.410562    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87addd1d-caad-4ed2-92e9-95ccd254ad6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692\" (UID: \"87addd1d-caad-4ed2-92e9-95ccd254ad6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692"
Jul 24 18:09:22.410599 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.410602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czkdd\" (UniqueName: \"kubernetes.io/projected/87addd1d-caad-4ed2-92e9-95ccd254ad6a-kube-api-access-czkdd\") pod \"payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692\" (UID: \"87addd1d-caad-4ed2-92e9-95ccd254ad6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692"
Jul 24 18:09:22.421806 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.421766    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"]
Jul 24 18:09:22.424995 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:09:22.424966    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ce080c_b649_4050_92c8_067a0a826ad6.slice/crio-81d80db5884e83175fb38bcd9c3f7fa6afadd70d6b246d631d98c6291777cfd8 WatchSource:0}: Error finding container 81d80db5884e83175fb38bcd9c3f7fa6afadd70d6b246d631d98c6291777cfd8: Status 404 returned error can't find the container with id 81d80db5884e83175fb38bcd9c3f7fa6afadd70d6b246d631d98c6291777cfd8
Jul 24 18:09:22.512197 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.512100    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87addd1d-caad-4ed2-92e9-95ccd254ad6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692\" (UID: \"87addd1d-caad-4ed2-92e9-95ccd254ad6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692"
Jul 24 18:09:22.512197 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.512139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czkdd\" (UniqueName: \"kubernetes.io/projected/87addd1d-caad-4ed2-92e9-95ccd254ad6a-kube-api-access-czkdd\") pod \"payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692\" (UID: \"87addd1d-caad-4ed2-92e9-95ccd254ad6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692"
Jul 24 18:09:22.512898 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.512876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87addd1d-caad-4ed2-92e9-95ccd254ad6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692\" (UID: \"87addd1d-caad-4ed2-92e9-95ccd254ad6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692"
Jul 24 18:09:22.520878 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.520848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czkdd\" (UniqueName: \"kubernetes.io/projected/87addd1d-caad-4ed2-92e9-95ccd254ad6a-kube-api-access-czkdd\") pod \"payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692\" (UID: \"87addd1d-caad-4ed2-92e9-95ccd254ad6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692"
Jul 24 18:09:22.562739 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.562704    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw" event={"ID":"90ce080c-b649-4050-92c8-067a0a826ad6","Type":"ContainerStarted","Data":"81d80db5884e83175fb38bcd9c3f7fa6afadd70d6b246d631d98c6291777cfd8"}
Jul 24 18:09:22.608385 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.608348    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692"
Jul 24 18:09:22.752527 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:22.752503    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692"]
Jul 24 18:09:22.754799 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:09:22.754752    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87addd1d_caad_4ed2_92e9_95ccd254ad6a.slice/crio-6581c585c64d41d46998006d7850af340a8d801304c0361902326f113b12f08e WatchSource:0}: Error finding container 6581c585c64d41d46998006d7850af340a8d801304c0361902326f113b12f08e: Status 404 returned error can't find the container with id 6581c585c64d41d46998006d7850af340a8d801304c0361902326f113b12f08e
Jul 24 18:09:23.570342 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:23.570310    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw" event={"ID":"90ce080c-b649-4050-92c8-067a0a826ad6","Type":"ContainerStarted","Data":"558f4967e6e86918eeaf1988a3ba90213eea20bf2fa71fe14373c998680fb36d"}
Jul 24 18:09:23.570826 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:23.570425    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"
Jul 24 18:09:23.571725 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:23.571702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692" event={"ID":"87addd1d-caad-4ed2-92e9-95ccd254ad6a","Type":"ContainerStarted","Data":"d2a159eae441e7464a3882adf70853ac59dee9043595a0b186cc392488e9c648"}
Jul 24 18:09:23.571915 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:23.571747    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692" event={"ID":"87addd1d-caad-4ed2-92e9-95ccd254ad6a","Type":"ContainerStarted","Data":"6581c585c64d41d46998006d7850af340a8d801304c0361902326f113b12f08e"}
Jul 24 18:09:23.571915 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:23.571800    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692"
Jul 24 18:09:23.590165 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:23.590113    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw" podStartSLOduration=2.138843748 podStartE2EDuration="2.590096101s" totalImagesPullingTime="451.252353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:09:23.588660188 +0000 UTC m=+1586.112018667" watchObservedRunningTime="2026-07-24 18:09:23.590096101 +0000 UTC m=+1586.113454579"
Jul 24 18:09:23.604988 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:23.604914    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692" podStartSLOduration=1.6048670029999998 podStartE2EDuration="1.604867003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:09:23.604371098 +0000 UTC m=+1586.127729577" watchObservedRunningTime="2026-07-24 18:09:23.604867003 +0000 UTC m=+1586.128225482"
Jul 24 18:09:29.582237 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:29.582202    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"
Jul 24 18:09:34.579478 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:34.579440    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692"
Jul 24 18:09:57.537806 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:57.537690    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"]
Jul 24 18:09:57.538684 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:57.537980    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw" podUID="90ce080c-b649-4050-92c8-067a0a826ad6" containerName="maas-api" containerID="cri-o://558f4967e6e86918eeaf1988a3ba90213eea20bf2fa71fe14373c998680fb36d" gracePeriod=30
Jul 24 18:09:57.582196 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:57.582168    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:57.614292 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:57.614249    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692"]
Jul 24 18:09:57.614626 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:57.614565    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692" podUID="87addd1d-caad-4ed2-92e9-95ccd254ad6a" containerName="payload-pre-processing" containerID="cri-o://d2a159eae441e7464a3882adf70853ac59dee9043595a0b186cc392488e9c648" gracePeriod=30
Jul 24 18:09:57.688935 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:57.688904    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:57.747870 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:57.747836    2578 generic.go:356] "Generic (PLEG): container finished" podID="90ce080c-b649-4050-92c8-067a0a826ad6" containerID="558f4967e6e86918eeaf1988a3ba90213eea20bf2fa71fe14373c998680fb36d" exitCode=0
Jul 24 18:09:57.748077 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:57.747907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw" event={"ID":"90ce080c-b649-4050-92c8-067a0a826ad6","Type":"ContainerDied","Data":"558f4967e6e86918eeaf1988a3ba90213eea20bf2fa71fe14373c998680fb36d"}
Jul 24 18:09:57.749831 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:57.749807    2578 generic.go:356] "Generic (PLEG): container finished" podID="87addd1d-caad-4ed2-92e9-95ccd254ad6a" containerID="d2a159eae441e7464a3882adf70853ac59dee9043595a0b186cc392488e9c648" exitCode=0
Jul 24 18:09:57.749977 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:57.749879    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692" event={"ID":"87addd1d-caad-4ed2-92e9-95ccd254ad6a","Type":"ContainerDied","Data":"d2a159eae441e7464a3882adf70853ac59dee9043595a0b186cc392488e9c648"}
Jul 24 18:09:57.782113 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:57.781726    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:57.866178 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:57.866149    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"
Jul 24 18:09:57.893320 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:57.893284    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692"
Jul 24 18:09:57.914955 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:57.914923    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ce080c-b649-4050-92c8-067a0a826ad6-openshift-service-ca\") pod \"90ce080c-b649-4050-92c8-067a0a826ad6\" (UID: \"90ce080c-b649-4050-92c8-067a0a826ad6\") "
Jul 24 18:09:57.915133 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:57.914998    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttg8n\" (UniqueName: \"kubernetes.io/projected/90ce080c-b649-4050-92c8-067a0a826ad6-kube-api-access-ttg8n\") pod \"90ce080c-b649-4050-92c8-067a0a826ad6\" (UID: \"90ce080c-b649-4050-92c8-067a0a826ad6\") "
Jul 24 18:09:57.915133 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:57.915048    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ce080c-b649-4050-92c8-067a0a826ad6-maas-api-tls\") pod \"90ce080c-b649-4050-92c8-067a0a826ad6\" (UID: \"90ce080c-b649-4050-92c8-067a0a826ad6\") "
Jul 24 18:09:57.915388 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:57.915348    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ce080c-b649-4050-92c8-067a0a826ad6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "90ce080c-b649-4050-92c8-067a0a826ad6" (UID: "90ce080c-b649-4050-92c8-067a0a826ad6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:09:57.917232 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:57.917207    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ce080c-b649-4050-92c8-067a0a826ad6-kube-api-access-ttg8n" (OuterVolumeSpecName: "kube-api-access-ttg8n") pod "90ce080c-b649-4050-92c8-067a0a826ad6" (UID: "90ce080c-b649-4050-92c8-067a0a826ad6"). InnerVolumeSpecName "kube-api-access-ttg8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:09:57.917316 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:57.917229    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ce080c-b649-4050-92c8-067a0a826ad6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "90ce080c-b649-4050-92c8-067a0a826ad6" (UID: "90ce080c-b649-4050-92c8-067a0a826ad6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:09:58.016125 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:58.016096    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czkdd\" (UniqueName: \"kubernetes.io/projected/87addd1d-caad-4ed2-92e9-95ccd254ad6a-kube-api-access-czkdd\") pod \"87addd1d-caad-4ed2-92e9-95ccd254ad6a\" (UID: \"87addd1d-caad-4ed2-92e9-95ccd254ad6a\") "
Jul 24 18:09:58.016295 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:58.016144    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87addd1d-caad-4ed2-92e9-95ccd254ad6a-plugins-config-volume\") pod \"87addd1d-caad-4ed2-92e9-95ccd254ad6a\" (UID: \"87addd1d-caad-4ed2-92e9-95ccd254ad6a\") "
Jul 24 18:09:58.016295 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:58.016289    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ce080c-b649-4050-92c8-067a0a826ad6-openshift-service-ca\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:09:58.016377 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:58.016300    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttg8n\" (UniqueName: \"kubernetes.io/projected/90ce080c-b649-4050-92c8-067a0a826ad6-kube-api-access-ttg8n\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:09:58.016377 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:58.016311    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ce080c-b649-4050-92c8-067a0a826ad6-maas-api-tls\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:09:58.016571 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:58.016546    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87addd1d-caad-4ed2-92e9-95ccd254ad6a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "87addd1d-caad-4ed2-92e9-95ccd254ad6a" (UID: "87addd1d-caad-4ed2-92e9-95ccd254ad6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:09:58.018244 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:58.018225    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87addd1d-caad-4ed2-92e9-95ccd254ad6a-kube-api-access-czkdd" (OuterVolumeSpecName: "kube-api-access-czkdd") pod "87addd1d-caad-4ed2-92e9-95ccd254ad6a" (UID: "87addd1d-caad-4ed2-92e9-95ccd254ad6a"). InnerVolumeSpecName "kube-api-access-czkdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:09:58.117447 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:58.117345    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czkdd\" (UniqueName: \"kubernetes.io/projected/87addd1d-caad-4ed2-92e9-95ccd254ad6a-kube-api-access-czkdd\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:09:58.117447 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:58.117373    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87addd1d-caad-4ed2-92e9-95ccd254ad6a-plugins-config-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:09:58.755874 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:58.755845    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"
Jul 24 18:09:58.755874 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:58.755863    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw" event={"ID":"90ce080c-b649-4050-92c8-067a0a826ad6","Type":"ContainerDied","Data":"81d80db5884e83175fb38bcd9c3f7fa6afadd70d6b246d631d98c6291777cfd8"}
Jul 24 18:09:58.756467 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:58.755923    2578 scope.go:117] "RemoveContainer" containerID="558f4967e6e86918eeaf1988a3ba90213eea20bf2fa71fe14373c998680fb36d"
Jul 24 18:09:58.757246 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:58.757221    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692" event={"ID":"87addd1d-caad-4ed2-92e9-95ccd254ad6a","Type":"ContainerDied","Data":"6581c585c64d41d46998006d7850af340a8d801304c0361902326f113b12f08e"}
Jul 24 18:09:58.757246 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:58.757235    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692"
Jul 24 18:09:58.766033 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:58.766001    2578 scope.go:117] "RemoveContainer" containerID="d2a159eae441e7464a3882adf70853ac59dee9043595a0b186cc392488e9c648"
Jul 24 18:09:58.775724 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:58.775700    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"]
Jul 24 18:09:58.777535 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:58.777515    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f37130a8-6b44c8c6b5-w6vgw"]
Jul 24 18:09:58.789854 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:58.789828    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692"]
Jul 24 18:09:58.795900 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:58.795878    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f37130a8-6bc9f7fd7b-92692"]
Jul 24 18:09:59.967150 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:59.967121    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87addd1d-caad-4ed2-92e9-95ccd254ad6a" path="/var/lib/kubelet/pods/87addd1d-caad-4ed2-92e9-95ccd254ad6a/volumes"
Jul 24 18:09:59.967517 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:09:59.967505    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90ce080c-b649-4050-92c8-067a0a826ad6" path="/var/lib/kubelet/pods/90ce080c-b649-4050-92c8-067a0a826ad6/volumes"
Jul 24 18:10:02.199989 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:02.199950    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:03.621600 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:03.621560    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:03.737639 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:03.737599    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:03.976990 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:03.976901    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:04.304955 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:04.304925    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:04.653184 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:04.653102    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:06.265327 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:06.265295    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:07.173726 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.173680    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:07.262171 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.262134    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:07.346225 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.346187    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"]
Jul 24 18:10:07.346766 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.346746    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90ce080c-b649-4050-92c8-067a0a826ad6" containerName="maas-api"
Jul 24 18:10:07.346864 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.346769    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="90ce080c-b649-4050-92c8-067a0a826ad6" containerName="maas-api"
Jul 24 18:10:07.346864 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.346803    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87addd1d-caad-4ed2-92e9-95ccd254ad6a" containerName="payload-pre-processing"
Jul 24 18:10:07.346864 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.346812    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="87addd1d-caad-4ed2-92e9-95ccd254ad6a" containerName="payload-pre-processing"
Jul 24 18:10:07.347035 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.346924    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="87addd1d-caad-4ed2-92e9-95ccd254ad6a" containerName="payload-pre-processing"
Jul 24 18:10:07.347035 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.346951    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="90ce080c-b649-4050-92c8-067a0a826ad6" containerName="maas-api"
Jul 24 18:10:07.351074 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.351053    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"
Jul 24 18:10:07.353831 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.353807    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-294b6e13-serving-cert\""
Jul 24 18:10:07.361742 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.361719    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"]
Jul 24 18:10:07.382927 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.382896    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:07.405222 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.405185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx6pz\" (UniqueName: \"kubernetes.io/projected/fb71e234-1025-4e9e-847f-cc0f49e4f916-kube-api-access-jx6pz\") pod \"maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz\" (UID: \"fb71e234-1025-4e9e-847f-cc0f49e4f916\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"
Jul 24 18:10:07.405392 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.405233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb71e234-1025-4e9e-847f-cc0f49e4f916-maas-api-tls\") pod \"maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz\" (UID: \"fb71e234-1025-4e9e-847f-cc0f49e4f916\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"
Jul 24 18:10:07.405392 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.405372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb71e234-1025-4e9e-847f-cc0f49e4f916-openshift-service-ca\") pod \"maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz\" (UID: \"fb71e234-1025-4e9e-847f-cc0f49e4f916\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"
Jul 24 18:10:07.506060 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.505968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb71e234-1025-4e9e-847f-cc0f49e4f916-openshift-service-ca\") pod \"maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz\" (UID: \"fb71e234-1025-4e9e-847f-cc0f49e4f916\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"
Jul 24 18:10:07.506217 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.506077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx6pz\" (UniqueName: \"kubernetes.io/projected/fb71e234-1025-4e9e-847f-cc0f49e4f916-kube-api-access-jx6pz\") pod \"maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz\" (UID: \"fb71e234-1025-4e9e-847f-cc0f49e4f916\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"
Jul 24 18:10:07.506217 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.506119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb71e234-1025-4e9e-847f-cc0f49e4f916-maas-api-tls\") pod \"maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz\" (UID: \"fb71e234-1025-4e9e-847f-cc0f49e4f916\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"
Jul 24 18:10:07.506309 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:10:07.506290    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-serving-cert: secret "maas-api-e2e-derive-294b6e13-serving-cert" not found
Jul 24 18:10:07.506372 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:10:07.506361    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb71e234-1025-4e9e-847f-cc0f49e4f916-maas-api-tls podName:fb71e234-1025-4e9e-847f-cc0f49e4f916 nodeName:}" failed. No retries permitted until 2026-07-24 18:10:08.006340764 +0000 UTC m=+1630.529699234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb71e234-1025-4e9e-847f-cc0f49e4f916-maas-api-tls") pod "maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz" (UID: "fb71e234-1025-4e9e-847f-cc0f49e4f916") : secret "maas-api-e2e-derive-294b6e13-serving-cert" not found
Jul 24 18:10:07.506766 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.506747    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb71e234-1025-4e9e-847f-cc0f49e4f916-openshift-service-ca\") pod \"maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz\" (UID: \"fb71e234-1025-4e9e-847f-cc0f49e4f916\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"
Jul 24 18:10:07.517196 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:07.517174    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx6pz\" (UniqueName: \"kubernetes.io/projected/fb71e234-1025-4e9e-847f-cc0f49e4f916-kube-api-access-jx6pz\") pod \"maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz\" (UID: \"fb71e234-1025-4e9e-847f-cc0f49e4f916\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"
Jul 24 18:10:08.006513 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.006476    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:08.010754 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.010727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb71e234-1025-4e9e-847f-cc0f49e4f916-maas-api-tls\") pod \"maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz\" (UID: \"fb71e234-1025-4e9e-847f-cc0f49e4f916\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"
Jul 24 18:10:08.013142 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.013109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb71e234-1025-4e9e-847f-cc0f49e4f916-maas-api-tls\") pod \"maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz\" (UID: \"fb71e234-1025-4e9e-847f-cc0f49e4f916\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"
Jul 24 18:10:08.061682 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.061653    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx"]
Jul 24 18:10:08.065238 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.065217    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx"
Jul 24 18:10:08.070190 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.070155    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-294b6e13\""
Jul 24 18:10:08.073150 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.070639    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-294b6e13-dockercfg-nrqhm\""
Jul 24 18:10:08.082888 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.082858    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx"]
Jul 24 18:10:08.086951 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.086922    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4"]
Jul 24 18:10:08.090136 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.090121    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4"
Jul 24 18:10:08.105399 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.105368    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4"]
Jul 24 18:10:08.108286 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.108262    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:08.212337 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.212301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f8e924-f900-4016-8be6-a799a9456e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx\" (UID: \"b9f8e924-f900-4016-8be6-a799a9456e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx"
Jul 24 18:10:08.212498 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.212347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6458c6c6-5b97-469e-9dea-37e1895002fc-plugins-config-volume\") pod \"payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4\" (UID: \"6458c6c6-5b97-469e-9dea-37e1895002fc\") " pod="openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4"
Jul 24 18:10:08.212498 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.212385    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjbbm\" (UniqueName: \"kubernetes.io/projected/6458c6c6-5b97-469e-9dea-37e1895002fc-kube-api-access-pjbbm\") pod \"payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4\" (UID: \"6458c6c6-5b97-469e-9dea-37e1895002fc\") " pod="openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4"
Jul 24 18:10:08.212580 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.212510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdn7z\" (UniqueName: \"kubernetes.io/projected/b9f8e924-f900-4016-8be6-a799a9456e9b-kube-api-access-bdn7z\") pod \"payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx\" (UID: \"b9f8e924-f900-4016-8be6-a799a9456e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx"
Jul 24 18:10:08.262973 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.262892    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"
Jul 24 18:10:08.313118 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.313082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdn7z\" (UniqueName: \"kubernetes.io/projected/b9f8e924-f900-4016-8be6-a799a9456e9b-kube-api-access-bdn7z\") pod \"payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx\" (UID: \"b9f8e924-f900-4016-8be6-a799a9456e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx"
Jul 24 18:10:08.313308 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.313147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f8e924-f900-4016-8be6-a799a9456e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx\" (UID: \"b9f8e924-f900-4016-8be6-a799a9456e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx"
Jul 24 18:10:08.313308 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.313183    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6458c6c6-5b97-469e-9dea-37e1895002fc-plugins-config-volume\") pod \"payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4\" (UID: \"6458c6c6-5b97-469e-9dea-37e1895002fc\") " pod="openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4"
Jul 24 18:10:08.313308 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.313219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjbbm\" (UniqueName: \"kubernetes.io/projected/6458c6c6-5b97-469e-9dea-37e1895002fc-kube-api-access-pjbbm\") pod \"payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4\" (UID: \"6458c6c6-5b97-469e-9dea-37e1895002fc\") " pod="openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4"
Jul 24 18:10:08.313972 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.313950    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f8e924-f900-4016-8be6-a799a9456e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx\" (UID: \"b9f8e924-f900-4016-8be6-a799a9456e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx"
Jul 24 18:10:08.314069 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.314048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6458c6c6-5b97-469e-9dea-37e1895002fc-plugins-config-volume\") pod \"payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4\" (UID: \"6458c6c6-5b97-469e-9dea-37e1895002fc\") " pod="openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4"
Jul 24 18:10:08.333521 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.333489    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjbbm\" (UniqueName: \"kubernetes.io/projected/6458c6c6-5b97-469e-9dea-37e1895002fc-kube-api-access-pjbbm\") pod \"payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4\" (UID: \"6458c6c6-5b97-469e-9dea-37e1895002fc\") " pod="openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4"
Jul 24 18:10:08.335907 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.335879    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdn7z\" (UniqueName: \"kubernetes.io/projected/b9f8e924-f900-4016-8be6-a799a9456e9b-kube-api-access-bdn7z\") pod \"payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx\" (UID: \"b9f8e924-f900-4016-8be6-a799a9456e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx"
Jul 24 18:10:08.383465 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.383432    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx"
Jul 24 18:10:08.404431 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.404395    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4"
Jul 24 18:10:08.411477 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.411448    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"]
Jul 24 18:10:08.415738 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:10:08.415710    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb71e234_1025_4e9e_847f_cc0f49e4f916.slice/crio-cb89f9f52db7663780d8052b512ffeb6451f8316083b8b9f32b62e6a3f965c25 WatchSource:0}: Error finding container cb89f9f52db7663780d8052b512ffeb6451f8316083b8b9f32b62e6a3f965c25: Status 404 returned error can't find the container with id cb89f9f52db7663780d8052b512ffeb6451f8316083b8b9f32b62e6a3f965c25
Jul 24 18:10:08.548891 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.548715    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx"]
Jul 24 18:10:08.551501 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:10:08.551469    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f8e924_f900_4016_8be6_a799a9456e9b.slice/crio-ce58fedcfec7ec3d1a10f7eee880b43d0c0619628489253167980297cbdd0e01 WatchSource:0}: Error finding container ce58fedcfec7ec3d1a10f7eee880b43d0c0619628489253167980297cbdd0e01: Status 404 returned error can't find the container with id ce58fedcfec7ec3d1a10f7eee880b43d0c0619628489253167980297cbdd0e01
Jul 24 18:10:08.569054 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.569028    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4"]
Jul 24 18:10:08.570816 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:10:08.570762    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6458c6c6_5b97_469e_9dea_37e1895002fc.slice/crio-c94a2ed7bf4c6c8d09d12260e6e9db3c0f3c3cad8b741a6215b48e681d9b76dc WatchSource:0}: Error finding container c94a2ed7bf4c6c8d09d12260e6e9db3c0f3c3cad8b741a6215b48e681d9b76dc: Status 404 returned error can't find the container with id c94a2ed7bf4c6c8d09d12260e6e9db3c0f3c3cad8b741a6215b48e681d9b76dc
Jul 24 18:10:08.811650 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.811564    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz" event={"ID":"fb71e234-1025-4e9e-847f-cc0f49e4f916","Type":"ContainerStarted","Data":"cb89f9f52db7663780d8052b512ffeb6451f8316083b8b9f32b62e6a3f965c25"}
Jul 24 18:10:08.813087 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.813056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4" event={"ID":"6458c6c6-5b97-469e-9dea-37e1895002fc","Type":"ContainerStarted","Data":"761b6fd9dfa27674aa81bf6a1cad49671d2b5975c99ecce315c91e2c6346d59d"}
Jul 24 18:10:08.813222 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.813106    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4" event={"ID":"6458c6c6-5b97-469e-9dea-37e1895002fc","Type":"ContainerStarted","Data":"c94a2ed7bf4c6c8d09d12260e6e9db3c0f3c3cad8b741a6215b48e681d9b76dc"}
Jul 24 18:10:08.813222 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.813178    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4"
Jul 24 18:10:08.814575 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.814552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx" event={"ID":"b9f8e924-f900-4016-8be6-a799a9456e9b","Type":"ContainerStarted","Data":"a3aa2b6173be52745852d2a3bde93ad1e1d02a997f7c2715ee39933048f1e801"}
Jul 24 18:10:08.814713 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.814597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx" event={"ID":"b9f8e924-f900-4016-8be6-a799a9456e9b","Type":"ContainerStarted","Data":"ce58fedcfec7ec3d1a10f7eee880b43d0c0619628489253167980297cbdd0e01"}
Jul 24 18:10:08.814713 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.814625    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx"
Jul 24 18:10:08.831935 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.831890    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4" podStartSLOduration=0.831875076 podStartE2EDuration="831.875076ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:10:08.829843276 +0000 UTC m=+1631.353201755" watchObservedRunningTime="2026-07-24 18:10:08.831875076 +0000 UTC m=+1631.355233532"
Jul 24 18:10:08.846747 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:08.846697    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx" podStartSLOduration=0.846684534 podStartE2EDuration="846.684534ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:10:08.844993705 +0000 UTC m=+1631.368352184" watchObservedRunningTime="2026-07-24 18:10:08.846684534 +0000 UTC m=+1631.370043012"
Jul 24 18:10:09.822117 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:09.822074    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz" event={"ID":"fb71e234-1025-4e9e-847f-cc0f49e4f916","Type":"ContainerStarted","Data":"fc960ffe94c19929504411f908bf0b11fb866bd572182824e180e83c4d666d28"}
Jul 24 18:10:09.822577 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:09.822457    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"
Jul 24 18:10:09.843210 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:09.843160    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz" podStartSLOduration=2.347739684 podStartE2EDuration="2.843144734s" totalImagesPullingTime="495.40505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:10:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:10:09.841389255 +0000 UTC m=+1632.364747734" watchObservedRunningTime="2026-07-24 18:10:09.843144734 +0000 UTC m=+1632.366503212"
Jul 24 18:10:15.832670 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:15.832636    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"
Jul 24 18:10:19.824899 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:19.824862    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4"
Jul 24 18:10:19.825296 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:19.824936    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx"
Jul 24 18:10:23.956192 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:23.956159    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:24.014358 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:24.014325    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:24.627095 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:24.627055    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:42.503237 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.503200    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"]
Jul 24 18:10:42.504835 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.504761    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz" podUID="fb71e234-1025-4e9e-847f-cc0f49e4f916" containerName="maas-api" containerID="cri-o://fc960ffe94c19929504411f908bf0b11fb866bd572182824e180e83c4d666d28" gracePeriod=30
Jul 24 18:10:42.537810 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.537748    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:42.561061 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.561021    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4"]
Jul 24 18:10:42.561336 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.561283    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4" podUID="6458c6c6-5b97-469e-9dea-37e1895002fc" containerName="payload-processing" containerID="cri-o://761b6fd9dfa27674aa81bf6a1cad49671d2b5975c99ecce315c91e2c6346d59d" gracePeriod=30
Jul 24 18:10:42.566115 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.566079    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx"]
Jul 24 18:10:42.566115 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.566306    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx" podUID="b9f8e924-f900-4016-8be6-a799a9456e9b" containerName="payload-pre-processing" containerID="cri-o://a3aa2b6173be52745852d2a3bde93ad1e1d02a997f7c2715ee39933048f1e801" gracePeriod=30
Jul 24 18:10:42.609819 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.609710    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:42.830308 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.830277    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"
Jul 24 18:10:42.873869 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.873837    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx"
Jul 24 18:10:42.877005 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.876980    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4"
Jul 24 18:10:42.958049 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.958016    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6458c6c6-5b97-469e-9dea-37e1895002fc-plugins-config-volume\") pod \"6458c6c6-5b97-469e-9dea-37e1895002fc\" (UID: \"6458c6c6-5b97-469e-9dea-37e1895002fc\") "
Jul 24 18:10:42.958245 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.958096    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx6pz\" (UniqueName: \"kubernetes.io/projected/fb71e234-1025-4e9e-847f-cc0f49e4f916-kube-api-access-jx6pz\") pod \"fb71e234-1025-4e9e-847f-cc0f49e4f916\" (UID: \"fb71e234-1025-4e9e-847f-cc0f49e4f916\") "
Jul 24 18:10:42.958245 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.958121    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb71e234-1025-4e9e-847f-cc0f49e4f916-openshift-service-ca\") pod \"fb71e234-1025-4e9e-847f-cc0f49e4f916\" (UID: \"fb71e234-1025-4e9e-847f-cc0f49e4f916\") "
Jul 24 18:10:42.958245 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.958186    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdn7z\" (UniqueName: \"kubernetes.io/projected/b9f8e924-f900-4016-8be6-a799a9456e9b-kube-api-access-bdn7z\") pod \"b9f8e924-f900-4016-8be6-a799a9456e9b\" (UID: \"b9f8e924-f900-4016-8be6-a799a9456e9b\") "
Jul 24 18:10:42.958245 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.958214    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f8e924-f900-4016-8be6-a799a9456e9b-plugins-config-volume\") pod \"b9f8e924-f900-4016-8be6-a799a9456e9b\" (UID: \"b9f8e924-f900-4016-8be6-a799a9456e9b\") "
Jul 24 18:10:42.958475 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.958265    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjbbm\" (UniqueName: \"kubernetes.io/projected/6458c6c6-5b97-469e-9dea-37e1895002fc-kube-api-access-pjbbm\") pod \"6458c6c6-5b97-469e-9dea-37e1895002fc\" (UID: \"6458c6c6-5b97-469e-9dea-37e1895002fc\") "
Jul 24 18:10:42.958475 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.958298    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb71e234-1025-4e9e-847f-cc0f49e4f916-maas-api-tls\") pod \"fb71e234-1025-4e9e-847f-cc0f49e4f916\" (UID: \"fb71e234-1025-4e9e-847f-cc0f49e4f916\") "
Jul 24 18:10:42.958602 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.958571    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6458c6c6-5b97-469e-9dea-37e1895002fc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6458c6c6-5b97-469e-9dea-37e1895002fc" (UID: "6458c6c6-5b97-469e-9dea-37e1895002fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:10:42.958672 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.958605    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb71e234-1025-4e9e-847f-cc0f49e4f916-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb71e234-1025-4e9e-847f-cc0f49e4f916" (UID: "fb71e234-1025-4e9e-847f-cc0f49e4f916"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:10:42.958672 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.958638    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f8e924-f900-4016-8be6-a799a9456e9b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9f8e924-f900-4016-8be6-a799a9456e9b" (UID: "b9f8e924-f900-4016-8be6-a799a9456e9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:10:42.960364 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.960332    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f8e924-f900-4016-8be6-a799a9456e9b-kube-api-access-bdn7z" (OuterVolumeSpecName: "kube-api-access-bdn7z") pod "b9f8e924-f900-4016-8be6-a799a9456e9b" (UID: "b9f8e924-f900-4016-8be6-a799a9456e9b"). InnerVolumeSpecName "kube-api-access-bdn7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:10:42.960461 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.960394    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6458c6c6-5b97-469e-9dea-37e1895002fc-kube-api-access-pjbbm" (OuterVolumeSpecName: "kube-api-access-pjbbm") pod "6458c6c6-5b97-469e-9dea-37e1895002fc" (UID: "6458c6c6-5b97-469e-9dea-37e1895002fc"). InnerVolumeSpecName "kube-api-access-pjbbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:10:42.960655 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.960634    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb71e234-1025-4e9e-847f-cc0f49e4f916-kube-api-access-jx6pz" (OuterVolumeSpecName: "kube-api-access-jx6pz") pod "fb71e234-1025-4e9e-847f-cc0f49e4f916" (UID: "fb71e234-1025-4e9e-847f-cc0f49e4f916"). InnerVolumeSpecName "kube-api-access-jx6pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:10:42.960751 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.960733    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb71e234-1025-4e9e-847f-cc0f49e4f916-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb71e234-1025-4e9e-847f-cc0f49e4f916" (UID: "fb71e234-1025-4e9e-847f-cc0f49e4f916"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:10:42.989323 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.989292    2578 generic.go:356] "Generic (PLEG): container finished" podID="fb71e234-1025-4e9e-847f-cc0f49e4f916" containerID="fc960ffe94c19929504411f908bf0b11fb866bd572182824e180e83c4d666d28" exitCode=0
Jul 24 18:10:42.989437 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.989342    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"
Jul 24 18:10:42.989437 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.989370    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz" event={"ID":"fb71e234-1025-4e9e-847f-cc0f49e4f916","Type":"ContainerDied","Data":"fc960ffe94c19929504411f908bf0b11fb866bd572182824e180e83c4d666d28"}
Jul 24 18:10:42.989437 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.989421    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz" event={"ID":"fb71e234-1025-4e9e-847f-cc0f49e4f916","Type":"ContainerDied","Data":"cb89f9f52db7663780d8052b512ffeb6451f8316083b8b9f32b62e6a3f965c25"}
Jul 24 18:10:42.989575 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.989445    2578 scope.go:117] "RemoveContainer" containerID="fc960ffe94c19929504411f908bf0b11fb866bd572182824e180e83c4d666d28"
Jul 24 18:10:42.990697 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.990676    2578 generic.go:356] "Generic (PLEG): container finished" podID="6458c6c6-5b97-469e-9dea-37e1895002fc" containerID="761b6fd9dfa27674aa81bf6a1cad49671d2b5975c99ecce315c91e2c6346d59d" exitCode=0
Jul 24 18:10:42.990800 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.990732    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4"
Jul 24 18:10:42.990800 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.990734    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4" event={"ID":"6458c6c6-5b97-469e-9dea-37e1895002fc","Type":"ContainerDied","Data":"761b6fd9dfa27674aa81bf6a1cad49671d2b5975c99ecce315c91e2c6346d59d"}
Jul 24 18:10:42.990926 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.990798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4" event={"ID":"6458c6c6-5b97-469e-9dea-37e1895002fc","Type":"ContainerDied","Data":"c94a2ed7bf4c6c8d09d12260e6e9db3c0f3c3cad8b741a6215b48e681d9b76dc"}
Jul 24 18:10:42.992191 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.992169    2578 generic.go:356] "Generic (PLEG): container finished" podID="b9f8e924-f900-4016-8be6-a799a9456e9b" containerID="a3aa2b6173be52745852d2a3bde93ad1e1d02a997f7c2715ee39933048f1e801" exitCode=0
Jul 24 18:10:42.992299 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.992197    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx" event={"ID":"b9f8e924-f900-4016-8be6-a799a9456e9b","Type":"ContainerDied","Data":"a3aa2b6173be52745852d2a3bde93ad1e1d02a997f7c2715ee39933048f1e801"}
Jul 24 18:10:42.992299 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.992225    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx"
Jul 24 18:10:42.992299 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:42.992236    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx" event={"ID":"b9f8e924-f900-4016-8be6-a799a9456e9b","Type":"ContainerDied","Data":"ce58fedcfec7ec3d1a10f7eee880b43d0c0619628489253167980297cbdd0e01"}
Jul 24 18:10:43.000558 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.000527    2578 scope.go:117] "RemoveContainer" containerID="fc960ffe94c19929504411f908bf0b11fb866bd572182824e180e83c4d666d28"
Jul 24 18:10:43.001063 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:10:43.000856    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc960ffe94c19929504411f908bf0b11fb866bd572182824e180e83c4d666d28\": container with ID starting with fc960ffe94c19929504411f908bf0b11fb866bd572182824e180e83c4d666d28 not found: ID does not exist" containerID="fc960ffe94c19929504411f908bf0b11fb866bd572182824e180e83c4d666d28"
Jul 24 18:10:43.001063 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.000891    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc960ffe94c19929504411f908bf0b11fb866bd572182824e180e83c4d666d28"} err="failed to get container status \"fc960ffe94c19929504411f908bf0b11fb866bd572182824e180e83c4d666d28\": rpc error: code = NotFound desc = could not find container \"fc960ffe94c19929504411f908bf0b11fb866bd572182824e180e83c4d666d28\": container with ID starting with fc960ffe94c19929504411f908bf0b11fb866bd572182824e180e83c4d666d28 not found: ID does not exist"
Jul 24 18:10:43.001063 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.000913    2578 scope.go:117] "RemoveContainer" containerID="761b6fd9dfa27674aa81bf6a1cad49671d2b5975c99ecce315c91e2c6346d59d"
Jul 24 18:10:43.010155 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.010138    2578 scope.go:117] "RemoveContainer" containerID="761b6fd9dfa27674aa81bf6a1cad49671d2b5975c99ecce315c91e2c6346d59d"
Jul 24 18:10:43.010413 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:10:43.010392    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"761b6fd9dfa27674aa81bf6a1cad49671d2b5975c99ecce315c91e2c6346d59d\": container with ID starting with 761b6fd9dfa27674aa81bf6a1cad49671d2b5975c99ecce315c91e2c6346d59d not found: ID does not exist" containerID="761b6fd9dfa27674aa81bf6a1cad49671d2b5975c99ecce315c91e2c6346d59d"
Jul 24 18:10:43.010486 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.010418    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761b6fd9dfa27674aa81bf6a1cad49671d2b5975c99ecce315c91e2c6346d59d"} err="failed to get container status \"761b6fd9dfa27674aa81bf6a1cad49671d2b5975c99ecce315c91e2c6346d59d\": rpc error: code = NotFound desc = could not find container \"761b6fd9dfa27674aa81bf6a1cad49671d2b5975c99ecce315c91e2c6346d59d\": container with ID starting with 761b6fd9dfa27674aa81bf6a1cad49671d2b5975c99ecce315c91e2c6346d59d not found: ID does not exist"
Jul 24 18:10:43.010486 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.010435    2578 scope.go:117] "RemoveContainer" containerID="a3aa2b6173be52745852d2a3bde93ad1e1d02a997f7c2715ee39933048f1e801"
Jul 24 18:10:43.017891 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.017870    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"]
Jul 24 18:10:43.018093 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.018076    2578 scope.go:117] "RemoveContainer" containerID="a3aa2b6173be52745852d2a3bde93ad1e1d02a997f7c2715ee39933048f1e801"
Jul 24 18:10:43.018338 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:10:43.018318    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3aa2b6173be52745852d2a3bde93ad1e1d02a997f7c2715ee39933048f1e801\": container with ID starting with a3aa2b6173be52745852d2a3bde93ad1e1d02a997f7c2715ee39933048f1e801 not found: ID does not exist" containerID="a3aa2b6173be52745852d2a3bde93ad1e1d02a997f7c2715ee39933048f1e801"
Jul 24 18:10:43.018383 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.018345    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3aa2b6173be52745852d2a3bde93ad1e1d02a997f7c2715ee39933048f1e801"} err="failed to get container status \"a3aa2b6173be52745852d2a3bde93ad1e1d02a997f7c2715ee39933048f1e801\": rpc error: code = NotFound desc = could not find container \"a3aa2b6173be52745852d2a3bde93ad1e1d02a997f7c2715ee39933048f1e801\": container with ID starting with a3aa2b6173be52745852d2a3bde93ad1e1d02a997f7c2715ee39933048f1e801 not found: ID does not exist"
Jul 24 18:10:43.022394 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.022370    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-294b6e13-fbf558fd7-9w2wz"]
Jul 24 18:10:43.033594 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.033544    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx"]
Jul 24 18:10:43.037614 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.037595    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-294b6e13-7b5fc9d48-sc7sx"]
Jul 24 18:10:43.048182 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.048162    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4"]
Jul 24 18:10:43.052274 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.052255    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-294b6e13-59789bcf74-dqnh4"]
Jul 24 18:10:43.059645 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.059629    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjbbm\" (UniqueName: \"kubernetes.io/projected/6458c6c6-5b97-469e-9dea-37e1895002fc-kube-api-access-pjbbm\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:10:43.059710 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.059647    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb71e234-1025-4e9e-847f-cc0f49e4f916-maas-api-tls\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:10:43.059710 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.059659    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6458c6c6-5b97-469e-9dea-37e1895002fc-plugins-config-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:10:43.059710 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.059667    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx6pz\" (UniqueName: \"kubernetes.io/projected/fb71e234-1025-4e9e-847f-cc0f49e4f916-kube-api-access-jx6pz\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:10:43.059710 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.059676    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb71e234-1025-4e9e-847f-cc0f49e4f916-openshift-service-ca\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:10:43.059710 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.059685    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdn7z\" (UniqueName: \"kubernetes.io/projected/b9f8e924-f900-4016-8be6-a799a9456e9b-kube-api-access-bdn7z\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:10:43.059710 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.059693    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f8e924-f900-4016-8be6-a799a9456e9b-plugins-config-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:10:43.967501 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.967467    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6458c6c6-5b97-469e-9dea-37e1895002fc" path="/var/lib/kubelet/pods/6458c6c6-5b97-469e-9dea-37e1895002fc/volumes"
Jul 24 18:10:43.967895 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.967829    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9f8e924-f900-4016-8be6-a799a9456e9b" path="/var/lib/kubelet/pods/b9f8e924-f900-4016-8be6-a799a9456e9b/volumes"
Jul 24 18:10:43.968185 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:43.968172    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb71e234-1025-4e9e-847f-cc0f49e4f916" path="/var/lib/kubelet/pods/fb71e234-1025-4e9e-847f-cc0f49e4f916/volumes"
Jul 24 18:10:47.155707 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:47.155673    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:47.323038 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:47.323000    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:47.605326 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:47.605291    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:48.076245 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:48.076205    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:48.175056 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:48.175011    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:48.278411 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:48.278373    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:49.797879 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:49.797838    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:56.533167 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:56.533116    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:56.635339 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:56.635298    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:56.733044 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:56.733008    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:56.834722 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:56.834694    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:57.261873 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:10:57.261754    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:23.321107 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:11:23.321061    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:29.107468 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:11:29.107423    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:29.212900 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:11:29.212866    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:29.304884 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:11:29.304848    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:45.295615 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:11:45.295579    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:51.191318 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:11:51.191282    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:51.303340 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:11:51.303305    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:51.407619 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:11:51.407574    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:02.015174 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:02.015131    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:02.902129 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:02.902093    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:09.778996 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:09.778962    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:09.960761 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:09.960727    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:10.056046 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:10.056015    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:10.923140 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:10.923107    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:11.050625 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:11.050593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:11.353845 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:11.353808    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:12.962969 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:12.962933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:18.808653 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:18.808618    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:20.081839 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:20.081806    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:20.189759 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:20.189725    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:20.816009 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:20.815970    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:26.628251 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:26.628219    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:26.738602 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:26.738568    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:26.880236 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:26.880148    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:27.228994 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:27.228912    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:27.691154 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:27.691108    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:30.507937 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.507895    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:30.605042 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.605003    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:30.617290 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.617255    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"]
Jul 24 18:12:30.617677 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.617665    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb71e234-1025-4e9e-847f-cc0f49e4f916" containerName="maas-api"
Jul 24 18:12:30.617727 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.617679    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb71e234-1025-4e9e-847f-cc0f49e4f916" containerName="maas-api"
Jul 24 18:12:30.617727 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.617689    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9f8e924-f900-4016-8be6-a799a9456e9b" containerName="payload-pre-processing"
Jul 24 18:12:30.617727 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.617696    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9f8e924-f900-4016-8be6-a799a9456e9b" containerName="payload-pre-processing"
Jul 24 18:12:30.617727 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.617707    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6458c6c6-5b97-469e-9dea-37e1895002fc" containerName="payload-processing"
Jul 24 18:12:30.617727 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.617713    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6458c6c6-5b97-469e-9dea-37e1895002fc" containerName="payload-processing"
Jul 24 18:12:30.617907 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.617775    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6458c6c6-5b97-469e-9dea-37e1895002fc" containerName="payload-processing"
Jul 24 18:12:30.617907 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.617795    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb71e234-1025-4e9e-847f-cc0f49e4f916" containerName="maas-api"
Jul 24 18:12:30.617907 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.617803    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9f8e924-f900-4016-8be6-a799a9456e9b" containerName="payload-pre-processing"
Jul 24 18:12:30.620977 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.620954    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"
Jul 24 18:12:30.623838 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.623813    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-72f7ddbd-serving-cert\""
Jul 24 18:12:30.637190 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.637160    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"]
Jul 24 18:12:30.683642 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.683606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd8mw\" (UniqueName: \"kubernetes.io/projected/60e776da-604b-45c4-bf0d-8bc9b719afd5-kube-api-access-cd8mw\") pod \"maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx\" (UID: \"60e776da-604b-45c4-bf0d-8bc9b719afd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"
Jul 24 18:12:30.683863 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.683652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60e776da-604b-45c4-bf0d-8bc9b719afd5-openshift-service-ca\") pod \"maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx\" (UID: \"60e776da-604b-45c4-bf0d-8bc9b719afd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"
Jul 24 18:12:30.683863 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.683839    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60e776da-604b-45c4-bf0d-8bc9b719afd5-maas-api-tls\") pod \"maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx\" (UID: \"60e776da-604b-45c4-bf0d-8bc9b719afd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"
Jul 24 18:12:30.714325 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.714295    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:30.784590 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.784508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd8mw\" (UniqueName: \"kubernetes.io/projected/60e776da-604b-45c4-bf0d-8bc9b719afd5-kube-api-access-cd8mw\") pod \"maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx\" (UID: \"60e776da-604b-45c4-bf0d-8bc9b719afd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"
Jul 24 18:12:30.784590 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.784552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60e776da-604b-45c4-bf0d-8bc9b719afd5-openshift-service-ca\") pod \"maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx\" (UID: \"60e776da-604b-45c4-bf0d-8bc9b719afd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"
Jul 24 18:12:30.784793 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.784624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60e776da-604b-45c4-bf0d-8bc9b719afd5-maas-api-tls\") pod \"maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx\" (UID: \"60e776da-604b-45c4-bf0d-8bc9b719afd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"
Jul 24 18:12:30.784793 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:12:30.784731    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-serving-cert: secret "maas-api-e2e-mt-72f7ddbd-serving-cert" not found
Jul 24 18:12:30.784898 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:12:30.784820    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60e776da-604b-45c4-bf0d-8bc9b719afd5-maas-api-tls podName:60e776da-604b-45c4-bf0d-8bc9b719afd5 nodeName:}" failed. No retries permitted until 2026-07-24 18:12:31.284802493 +0000 UTC m=+1773.808160965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60e776da-604b-45c4-bf0d-8bc9b719afd5-maas-api-tls") pod "maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx" (UID: "60e776da-604b-45c4-bf0d-8bc9b719afd5") : secret "maas-api-e2e-mt-72f7ddbd-serving-cert" not found
Jul 24 18:12:30.785276 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.785256    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60e776da-604b-45c4-bf0d-8bc9b719afd5-openshift-service-ca\") pod \"maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx\" (UID: \"60e776da-604b-45c4-bf0d-8bc9b719afd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"
Jul 24 18:12:30.796599 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.796569    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd8mw\" (UniqueName: \"kubernetes.io/projected/60e776da-604b-45c4-bf0d-8bc9b719afd5-kube-api-access-cd8mw\") pod \"maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx\" (UID: \"60e776da-604b-45c4-bf0d-8bc9b719afd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"
Jul 24 18:12:30.895714 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:30.895677    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:31.288921 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.288879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60e776da-604b-45c4-bf0d-8bc9b719afd5-maas-api-tls\") pod \"maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx\" (UID: \"60e776da-604b-45c4-bf0d-8bc9b719afd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"
Jul 24 18:12:31.291294 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.291267    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60e776da-604b-45c4-bf0d-8bc9b719afd5-maas-api-tls\") pod \"maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx\" (UID: \"60e776da-604b-45c4-bf0d-8bc9b719afd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"
Jul 24 18:12:31.330126 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.330096    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:31.516930 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.516894    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7"]
Jul 24 18:12:31.520479 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.520461    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7"
Jul 24 18:12:31.525542 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.525522    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-72f7ddbd\""
Jul 24 18:12:31.525804 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.525765    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-72f7ddbd-dockercfg-cx5h8\""
Jul 24 18:12:31.530686 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.530669    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"
Jul 24 18:12:31.561092 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.561059    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx"]
Jul 24 18:12:31.564632 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.564610    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx"
Jul 24 18:12:31.569835 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.569806    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:31.582183 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.582147    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx"]
Jul 24 18:12:31.593480 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.593448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjzwr\" (UniqueName: \"kubernetes.io/projected/9142865d-ca1e-4957-a513-4e76ffd3d9a1-kube-api-access-xjzwr\") pod \"payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7\" (UID: \"9142865d-ca1e-4957-a513-4e76ffd3d9a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7"
Jul 24 18:12:31.594434 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.594403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9142865d-ca1e-4957-a513-4e76ffd3d9a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7\" (UID: \"9142865d-ca1e-4957-a513-4e76ffd3d9a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7"
Jul 24 18:12:31.598422 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.598198    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7"]
Jul 24 18:12:31.688356 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.688330    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"]
Jul 24 18:12:31.691071 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:12:31.691042    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e776da_604b_45c4_bf0d_8bc9b719afd5.slice/crio-41ff395ed1fc5f505a02f1782b5c15e80300878ba34311b0cc3889930b49fecf WatchSource:0}: Error finding container 41ff395ed1fc5f505a02f1782b5c15e80300878ba34311b0cc3889930b49fecf: Status 404 returned error can't find the container with id 41ff395ed1fc5f505a02f1782b5c15e80300878ba34311b0cc3889930b49fecf
Jul 24 18:12:31.695841 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.695807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jx2d\" (UniqueName: \"kubernetes.io/projected/9426df1f-93ee-4a12-8464-15fde8d1e1a2-kube-api-access-2jx2d\") pod \"payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx\" (UID: \"9426df1f-93ee-4a12-8464-15fde8d1e1a2\") " pod="openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx"
Jul 24 18:12:31.695953 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.695929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9142865d-ca1e-4957-a513-4e76ffd3d9a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7\" (UID: \"9142865d-ca1e-4957-a513-4e76ffd3d9a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7"
Jul 24 18:12:31.696089 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.696039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjzwr\" (UniqueName: \"kubernetes.io/projected/9142865d-ca1e-4957-a513-4e76ffd3d9a1-kube-api-access-xjzwr\") pod \"payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7\" (UID: \"9142865d-ca1e-4957-a513-4e76ffd3d9a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7"
Jul 24 18:12:31.696142 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.696089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9426df1f-93ee-4a12-8464-15fde8d1e1a2-plugins-config-volume\") pod \"payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx\" (UID: \"9426df1f-93ee-4a12-8464-15fde8d1e1a2\") " pod="openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx"
Jul 24 18:12:31.696662 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.696637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9142865d-ca1e-4957-a513-4e76ffd3d9a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7\" (UID: \"9142865d-ca1e-4957-a513-4e76ffd3d9a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7"
Jul 24 18:12:31.708873 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.708844    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjzwr\" (UniqueName: \"kubernetes.io/projected/9142865d-ca1e-4957-a513-4e76ffd3d9a1-kube-api-access-xjzwr\") pod \"payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7\" (UID: \"9142865d-ca1e-4957-a513-4e76ffd3d9a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7"
Jul 24 18:12:31.808795 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.803242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9426df1f-93ee-4a12-8464-15fde8d1e1a2-plugins-config-volume\") pod \"payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx\" (UID: \"9426df1f-93ee-4a12-8464-15fde8d1e1a2\") " pod="openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx"
Jul 24 18:12:31.808795 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.803332    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jx2d\" (UniqueName: \"kubernetes.io/projected/9426df1f-93ee-4a12-8464-15fde8d1e1a2-kube-api-access-2jx2d\") pod \"payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx\" (UID: \"9426df1f-93ee-4a12-8464-15fde8d1e1a2\") " pod="openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx"
Jul 24 18:12:31.808795 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.804537    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9426df1f-93ee-4a12-8464-15fde8d1e1a2-plugins-config-volume\") pod \"payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx\" (UID: \"9426df1f-93ee-4a12-8464-15fde8d1e1a2\") " pod="openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx"
Jul 24 18:12:31.812839 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.812804    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jx2d\" (UniqueName: \"kubernetes.io/projected/9426df1f-93ee-4a12-8464-15fde8d1e1a2-kube-api-access-2jx2d\") pod \"payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx\" (UID: \"9426df1f-93ee-4a12-8464-15fde8d1e1a2\") " pod="openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx"
Jul 24 18:12:31.830668 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.830640    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7"
Jul 24 18:12:31.875311 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.875280    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx"
Jul 24 18:12:31.979369 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:31.979318    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7"]
Jul 24 18:12:31.981602 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:12:31.981563    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9142865d_ca1e_4957_a513_4e76ffd3d9a1.slice/crio-dd9a66351595c89ec11c26b661d99b0afaef8358c3d815c038be7c452b5fecb9 WatchSource:0}: Error finding container dd9a66351595c89ec11c26b661d99b0afaef8358c3d815c038be7c452b5fecb9: Status 404 returned error can't find the container with id dd9a66351595c89ec11c26b661d99b0afaef8358c3d815c038be7c452b5fecb9
Jul 24 18:12:32.030954 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:32.030917    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx"]
Jul 24 18:12:32.033926 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:12:32.033902    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9426df1f_93ee_4a12_8464_15fde8d1e1a2.slice/crio-969a10dd0b5a3f0fdc8fe22f645f7c02e09b608b7e6010a30198916d855e025e WatchSource:0}: Error finding container 969a10dd0b5a3f0fdc8fe22f645f7c02e09b608b7e6010a30198916d855e025e: Status 404 returned error can't find the container with id 969a10dd0b5a3f0fdc8fe22f645f7c02e09b608b7e6010a30198916d855e025e
Jul 24 18:12:32.554224 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:32.554187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx" event={"ID":"60e776da-604b-45c4-bf0d-8bc9b719afd5","Type":"ContainerStarted","Data":"754c10c2b91715df9b0c32d7688fb39570a3c5a8e42cd28f9c1c3be0f5a8d1b3"}
Jul 24 18:12:32.554665 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:32.554245    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx" event={"ID":"60e776da-604b-45c4-bf0d-8bc9b719afd5","Type":"ContainerStarted","Data":"41ff395ed1fc5f505a02f1782b5c15e80300878ba34311b0cc3889930b49fecf"}
Jul 24 18:12:32.554665 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:32.554333    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"
Jul 24 18:12:32.555741 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:32.555716    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx" event={"ID":"9426df1f-93ee-4a12-8464-15fde8d1e1a2","Type":"ContainerStarted","Data":"e4300416e2ef42094a64371ef15df6720fde101e551afd18521b09a926510330"}
Jul 24 18:12:32.555873 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:32.555761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx" event={"ID":"9426df1f-93ee-4a12-8464-15fde8d1e1a2","Type":"ContainerStarted","Data":"969a10dd0b5a3f0fdc8fe22f645f7c02e09b608b7e6010a30198916d855e025e"}
Jul 24 18:12:32.555873 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:32.555805    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx"
Jul 24 18:12:32.557099 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:32.557080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7" event={"ID":"9142865d-ca1e-4957-a513-4e76ffd3d9a1","Type":"ContainerStarted","Data":"a90d2d87520f15205171860e7e1443047e0d9d4548ccc31e21de3e5e32408190"}
Jul 24 18:12:32.557203 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:32.557116    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7" event={"ID":"9142865d-ca1e-4957-a513-4e76ffd3d9a1","Type":"ContainerStarted","Data":"dd9a66351595c89ec11c26b661d99b0afaef8358c3d815c038be7c452b5fecb9"}
Jul 24 18:12:32.557203 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:32.557140    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7"
Jul 24 18:12:32.571676 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:32.571619    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx" podStartSLOduration=2.101871476 podStartE2EDuration="2.571602663s" totalImagesPullingTime="469.731187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:12:32.571038629 +0000 UTC m=+1775.094397108" watchObservedRunningTime="2026-07-24 18:12:32.571602663 +0000 UTC m=+1775.094961142"
Jul 24 18:12:32.589284 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:32.589234    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7" podStartSLOduration=1.589218152 podStartE2EDuration="1.589218152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:12:32.587205587 +0000 UTC m=+1775.110564068" watchObservedRunningTime="2026-07-24 18:12:32.589218152 +0000 UTC m=+1775.112576676"
Jul 24 18:12:32.605096 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:32.605033    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx" podStartSLOduration=1.605014663 podStartE2EDuration="1.605014663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:12:32.603502575 +0000 UTC m=+1775.126861054" watchObservedRunningTime="2026-07-24 18:12:32.605014663 +0000 UTC m=+1775.128373125"
Jul 24 18:12:35.035048 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:35.035012    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:35.721362 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:35.721329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:35.820014 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:35.819982    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:36.182742 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:36.182707    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:38.568449 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:38.568399    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"
Jul 24 18:12:42.147146 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:42.147111    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:42.225974 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:42.225945    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:43.564459 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:43.564430    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7"
Jul 24 18:12:43.564978 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:43.564509    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx"
Jul 24 18:12:52.961714 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:52.961623    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"]
Jul 24 18:12:52.962319 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:52.961855    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx" podUID="60e776da-604b-45c4-bf0d-8bc9b719afd5" containerName="maas-api" containerID="cri-o://754c10c2b91715df9b0c32d7688fb39570a3c5a8e42cd28f9c1c3be0f5a8d1b3" gracePeriod=30
Jul 24 18:12:52.998218 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:52.998184    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:53.023833 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.023799    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx"]
Jul 24 18:12:53.024084 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.024056    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx" podUID="9426df1f-93ee-4a12-8464-15fde8d1e1a2" containerName="payload-processing" containerID="cri-o://e4300416e2ef42094a64371ef15df6720fde101e551afd18521b09a926510330" gracePeriod=30
Jul 24 18:12:53.034368 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.034343    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7"]
Jul 24 18:12:53.034649 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.034621    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7" podUID="9142865d-ca1e-4957-a513-4e76ffd3d9a1" containerName="payload-pre-processing" containerID="cri-o://a90d2d87520f15205171860e7e1443047e0d9d4548ccc31e21de3e5e32408190" gracePeriod=30
Jul 24 18:12:53.101129 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.101095    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:53.201158 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.201119    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:53.301879 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.301852    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"
Jul 24 18:12:53.333547 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.333516    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx"
Jul 24 18:12:53.341256 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.341234    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7"
Jul 24 18:12:53.411456 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.411425    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cd8mw\" (UniqueName: \"kubernetes.io/projected/60e776da-604b-45c4-bf0d-8bc9b719afd5-kube-api-access-cd8mw\") pod \"60e776da-604b-45c4-bf0d-8bc9b719afd5\" (UID: \"60e776da-604b-45c4-bf0d-8bc9b719afd5\") "
Jul 24 18:12:53.411456 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.411459    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9142865d-ca1e-4957-a513-4e76ffd3d9a1-plugins-config-volume\") pod \"9142865d-ca1e-4957-a513-4e76ffd3d9a1\" (UID: \"9142865d-ca1e-4957-a513-4e76ffd3d9a1\") "
Jul 24 18:12:53.411702 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.411503    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60e776da-604b-45c4-bf0d-8bc9b719afd5-openshift-service-ca\") pod \"60e776da-604b-45c4-bf0d-8bc9b719afd5\" (UID: \"60e776da-604b-45c4-bf0d-8bc9b719afd5\") "
Jul 24 18:12:53.411702 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.411587    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jx2d\" (UniqueName: \"kubernetes.io/projected/9426df1f-93ee-4a12-8464-15fde8d1e1a2-kube-api-access-2jx2d\") pod \"9426df1f-93ee-4a12-8464-15fde8d1e1a2\" (UID: \"9426df1f-93ee-4a12-8464-15fde8d1e1a2\") "
Jul 24 18:12:53.411702 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.411637    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9426df1f-93ee-4a12-8464-15fde8d1e1a2-plugins-config-volume\") pod \"9426df1f-93ee-4a12-8464-15fde8d1e1a2\" (UID: \"9426df1f-93ee-4a12-8464-15fde8d1e1a2\") "
Jul 24 18:12:53.411702 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.411672    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60e776da-604b-45c4-bf0d-8bc9b719afd5-maas-api-tls\") pod \"60e776da-604b-45c4-bf0d-8bc9b719afd5\" (UID: \"60e776da-604b-45c4-bf0d-8bc9b719afd5\") "
Jul 24 18:12:53.412038 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.411864    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9142865d-ca1e-4957-a513-4e76ffd3d9a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9142865d-ca1e-4957-a513-4e76ffd3d9a1" (UID: "9142865d-ca1e-4957-a513-4e76ffd3d9a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:12:53.412038 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.411949    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e776da-604b-45c4-bf0d-8bc9b719afd5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "60e776da-604b-45c4-bf0d-8bc9b719afd5" (UID: "60e776da-604b-45c4-bf0d-8bc9b719afd5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:12:53.412172 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.412051    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjzwr\" (UniqueName: \"kubernetes.io/projected/9142865d-ca1e-4957-a513-4e76ffd3d9a1-kube-api-access-xjzwr\") pod \"9142865d-ca1e-4957-a513-4e76ffd3d9a1\" (UID: \"9142865d-ca1e-4957-a513-4e76ffd3d9a1\") "
Jul 24 18:12:53.412172 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.412124    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9426df1f-93ee-4a12-8464-15fde8d1e1a2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9426df1f-93ee-4a12-8464-15fde8d1e1a2" (UID: "9426df1f-93ee-4a12-8464-15fde8d1e1a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:12:53.412451 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.412428    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9142865d-ca1e-4957-a513-4e76ffd3d9a1-plugins-config-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:12:53.412548 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.412454    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60e776da-604b-45c4-bf0d-8bc9b719afd5-openshift-service-ca\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:12:53.412548 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.412468    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9426df1f-93ee-4a12-8464-15fde8d1e1a2-plugins-config-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:12:53.413760 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.413735    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e776da-604b-45c4-bf0d-8bc9b719afd5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "60e776da-604b-45c4-bf0d-8bc9b719afd5" (UID: "60e776da-604b-45c4-bf0d-8bc9b719afd5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:12:53.413760 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.413738    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e776da-604b-45c4-bf0d-8bc9b719afd5-kube-api-access-cd8mw" (OuterVolumeSpecName: "kube-api-access-cd8mw") pod "60e776da-604b-45c4-bf0d-8bc9b719afd5" (UID: "60e776da-604b-45c4-bf0d-8bc9b719afd5"). InnerVolumeSpecName "kube-api-access-cd8mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:12:53.414003 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.413839    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9426df1f-93ee-4a12-8464-15fde8d1e1a2-kube-api-access-2jx2d" (OuterVolumeSpecName: "kube-api-access-2jx2d") pod "9426df1f-93ee-4a12-8464-15fde8d1e1a2" (UID: "9426df1f-93ee-4a12-8464-15fde8d1e1a2"). InnerVolumeSpecName "kube-api-access-2jx2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:12:53.414003 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.413942    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9142865d-ca1e-4957-a513-4e76ffd3d9a1-kube-api-access-xjzwr" (OuterVolumeSpecName: "kube-api-access-xjzwr") pod "9142865d-ca1e-4957-a513-4e76ffd3d9a1" (UID: "9142865d-ca1e-4957-a513-4e76ffd3d9a1"). InnerVolumeSpecName "kube-api-access-xjzwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:12:53.513662 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.513576    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jx2d\" (UniqueName: \"kubernetes.io/projected/9426df1f-93ee-4a12-8464-15fde8d1e1a2-kube-api-access-2jx2d\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:12:53.513662 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.513607    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60e776da-604b-45c4-bf0d-8bc9b719afd5-maas-api-tls\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:12:53.513662 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.513619    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjzwr\" (UniqueName: \"kubernetes.io/projected/9142865d-ca1e-4957-a513-4e76ffd3d9a1-kube-api-access-xjzwr\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:12:53.513662 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.513629    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd8mw\" (UniqueName: \"kubernetes.io/projected/60e776da-604b-45c4-bf0d-8bc9b719afd5-kube-api-access-cd8mw\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:12:53.667516 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.667476    2578 generic.go:356] "Generic (PLEG): container finished" podID="60e776da-604b-45c4-bf0d-8bc9b719afd5" containerID="754c10c2b91715df9b0c32d7688fb39570a3c5a8e42cd28f9c1c3be0f5a8d1b3" exitCode=0
Jul 24 18:12:53.667715 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.667520    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"
Jul 24 18:12:53.667715 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.667562    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx" event={"ID":"60e776da-604b-45c4-bf0d-8bc9b719afd5","Type":"ContainerDied","Data":"754c10c2b91715df9b0c32d7688fb39570a3c5a8e42cd28f9c1c3be0f5a8d1b3"}
Jul 24 18:12:53.667715 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.667629    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx" event={"ID":"60e776da-604b-45c4-bf0d-8bc9b719afd5","Type":"ContainerDied","Data":"41ff395ed1fc5f505a02f1782b5c15e80300878ba34311b0cc3889930b49fecf"}
Jul 24 18:12:53.667715 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.667660    2578 scope.go:117] "RemoveContainer" containerID="754c10c2b91715df9b0c32d7688fb39570a3c5a8e42cd28f9c1c3be0f5a8d1b3"
Jul 24 18:12:53.668806 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.668762    2578 generic.go:356] "Generic (PLEG): container finished" podID="9426df1f-93ee-4a12-8464-15fde8d1e1a2" containerID="e4300416e2ef42094a64371ef15df6720fde101e551afd18521b09a926510330" exitCode=0
Jul 24 18:12:53.668894 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.668819    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx"
Jul 24 18:12:53.668957 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.668816    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx" event={"ID":"9426df1f-93ee-4a12-8464-15fde8d1e1a2","Type":"ContainerDied","Data":"e4300416e2ef42094a64371ef15df6720fde101e551afd18521b09a926510330"}
Jul 24 18:12:53.669010 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.668959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx" event={"ID":"9426df1f-93ee-4a12-8464-15fde8d1e1a2","Type":"ContainerDied","Data":"969a10dd0b5a3f0fdc8fe22f645f7c02e09b608b7e6010a30198916d855e025e"}
Jul 24 18:12:53.670184 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.670159    2578 generic.go:356] "Generic (PLEG): container finished" podID="9142865d-ca1e-4957-a513-4e76ffd3d9a1" containerID="a90d2d87520f15205171860e7e1443047e0d9d4548ccc31e21de3e5e32408190" exitCode=0
Jul 24 18:12:53.670289 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.670208    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7"
Jul 24 18:12:53.670289 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.670238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7" event={"ID":"9142865d-ca1e-4957-a513-4e76ffd3d9a1","Type":"ContainerDied","Data":"a90d2d87520f15205171860e7e1443047e0d9d4548ccc31e21de3e5e32408190"}
Jul 24 18:12:53.670289 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.670285    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7" event={"ID":"9142865d-ca1e-4957-a513-4e76ffd3d9a1","Type":"ContainerDied","Data":"dd9a66351595c89ec11c26b661d99b0afaef8358c3d815c038be7c452b5fecb9"}
Jul 24 18:12:53.678419 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.678319    2578 scope.go:117] "RemoveContainer" containerID="754c10c2b91715df9b0c32d7688fb39570a3c5a8e42cd28f9c1c3be0f5a8d1b3"
Jul 24 18:12:53.678650 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:12:53.678629    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"754c10c2b91715df9b0c32d7688fb39570a3c5a8e42cd28f9c1c3be0f5a8d1b3\": container with ID starting with 754c10c2b91715df9b0c32d7688fb39570a3c5a8e42cd28f9c1c3be0f5a8d1b3 not found: ID does not exist" containerID="754c10c2b91715df9b0c32d7688fb39570a3c5a8e42cd28f9c1c3be0f5a8d1b3"
Jul 24 18:12:53.678715 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.678656    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"754c10c2b91715df9b0c32d7688fb39570a3c5a8e42cd28f9c1c3be0f5a8d1b3"} err="failed to get container status \"754c10c2b91715df9b0c32d7688fb39570a3c5a8e42cd28f9c1c3be0f5a8d1b3\": rpc error: code = NotFound desc = could not find container \"754c10c2b91715df9b0c32d7688fb39570a3c5a8e42cd28f9c1c3be0f5a8d1b3\": container with ID starting with 754c10c2b91715df9b0c32d7688fb39570a3c5a8e42cd28f9c1c3be0f5a8d1b3 not found: ID does not exist"
Jul 24 18:12:53.678715 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.678672    2578 scope.go:117] "RemoveContainer" containerID="e4300416e2ef42094a64371ef15df6720fde101e551afd18521b09a926510330"
Jul 24 18:12:53.686695 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.686677    2578 scope.go:117] "RemoveContainer" containerID="e4300416e2ef42094a64371ef15df6720fde101e551afd18521b09a926510330"
Jul 24 18:12:53.686979 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:12:53.686960    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4300416e2ef42094a64371ef15df6720fde101e551afd18521b09a926510330\": container with ID starting with e4300416e2ef42094a64371ef15df6720fde101e551afd18521b09a926510330 not found: ID does not exist" containerID="e4300416e2ef42094a64371ef15df6720fde101e551afd18521b09a926510330"
Jul 24 18:12:53.687035 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.686986    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4300416e2ef42094a64371ef15df6720fde101e551afd18521b09a926510330"} err="failed to get container status \"e4300416e2ef42094a64371ef15df6720fde101e551afd18521b09a926510330\": rpc error: code = NotFound desc = could not find container \"e4300416e2ef42094a64371ef15df6720fde101e551afd18521b09a926510330\": container with ID starting with e4300416e2ef42094a64371ef15df6720fde101e551afd18521b09a926510330 not found: ID does not exist"
Jul 24 18:12:53.687035 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.687001    2578 scope.go:117] "RemoveContainer" containerID="a90d2d87520f15205171860e7e1443047e0d9d4548ccc31e21de3e5e32408190"
Jul 24 18:12:53.695012 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.694987    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"]
Jul 24 18:12:53.697168 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.697105    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-72f7ddbd-7b9558f487-bvwpx"]
Jul 24 18:12:53.700109 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.700091    2578 scope.go:117] "RemoveContainer" containerID="a90d2d87520f15205171860e7e1443047e0d9d4548ccc31e21de3e5e32408190"
Jul 24 18:12:53.700712 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:12:53.700685    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90d2d87520f15205171860e7e1443047e0d9d4548ccc31e21de3e5e32408190\": container with ID starting with a90d2d87520f15205171860e7e1443047e0d9d4548ccc31e21de3e5e32408190 not found: ID does not exist" containerID="a90d2d87520f15205171860e7e1443047e0d9d4548ccc31e21de3e5e32408190"
Jul 24 18:12:53.700821 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.700722    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90d2d87520f15205171860e7e1443047e0d9d4548ccc31e21de3e5e32408190"} err="failed to get container status \"a90d2d87520f15205171860e7e1443047e0d9d4548ccc31e21de3e5e32408190\": rpc error: code = NotFound desc = could not find container \"a90d2d87520f15205171860e7e1443047e0d9d4548ccc31e21de3e5e32408190\": container with ID starting with a90d2d87520f15205171860e7e1443047e0d9d4548ccc31e21de3e5e32408190 not found: ID does not exist"
Jul 24 18:12:53.708901 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.708879    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7"]
Jul 24 18:12:53.712277 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.712257    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-72f7ddbd-7f58685944-8xjs7"]
Jul 24 18:12:53.723330 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.723308    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx"]
Jul 24 18:12:53.725297 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.725278    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-72f7ddbd-75c7ddddf8-zrfvx"]
Jul 24 18:12:53.967655 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.967622    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60e776da-604b-45c4-bf0d-8bc9b719afd5" path="/var/lib/kubelet/pods/60e776da-604b-45c4-bf0d-8bc9b719afd5/volumes"
Jul 24 18:12:53.968047 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.968032    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9142865d-ca1e-4957-a513-4e76ffd3d9a1" path="/var/lib/kubelet/pods/9142865d-ca1e-4957-a513-4e76ffd3d9a1/volumes"
Jul 24 18:12:53.968361 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:53.968350    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9426df1f-93ee-4a12-8464-15fde8d1e1a2" path="/var/lib/kubelet/pods/9426df1f-93ee-4a12-8464-15fde8d1e1a2/volumes"
Jul 24 18:12:57.598246 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:57.598197    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:58.031047 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:12:58.030962    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:04.035193 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:04.035160    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:04.137760 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:04.137721    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:04.323234 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:04.323204    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:04.552849 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:04.552815    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:04.967649 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:04.967620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:05.035795 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:05.035743    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:11.002620 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:11.002572    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:11.113025 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:11.112988    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:11.205045 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:11.205012    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:11.665425 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:11.665394    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:17.158908 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:17.158871    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:17.260234 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:17.260202    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:24.068880 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:24.068850    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:24.492548 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:24.492451    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:24.679999 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:24.679965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:27.837045 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:27.837014    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:27.927391 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:27.927358    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"]
Jul 24 18:13:27.927803 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:27.927773    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9426df1f-93ee-4a12-8464-15fde8d1e1a2" containerName="payload-processing"
Jul 24 18:13:27.927854 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:27.927805    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9426df1f-93ee-4a12-8464-15fde8d1e1a2" containerName="payload-processing"
Jul 24 18:13:27.927854 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:27.927817    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9142865d-ca1e-4957-a513-4e76ffd3d9a1" containerName="payload-pre-processing"
Jul 24 18:13:27.927854 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:27.927823    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9142865d-ca1e-4957-a513-4e76ffd3d9a1" containerName="payload-pre-processing"
Jul 24 18:13:27.927854 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:27.927848    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60e776da-604b-45c4-bf0d-8bc9b719afd5" containerName="maas-api"
Jul 24 18:13:27.927854 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:27.927855    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="60e776da-604b-45c4-bf0d-8bc9b719afd5" containerName="maas-api"
Jul 24 18:13:27.928013 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:27.927907    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="60e776da-604b-45c4-bf0d-8bc9b719afd5" containerName="maas-api"
Jul 24 18:13:27.928013 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:27.927914    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9142865d-ca1e-4957-a513-4e76ffd3d9a1" containerName="payload-pre-processing"
Jul 24 18:13:27.928013 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:27.927922    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9426df1f-93ee-4a12-8464-15fde8d1e1a2" containerName="payload-processing"
Jul 24 18:13:27.931337 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:27.931308    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"
Jul 24 18:13:27.932468 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:27.932444    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:27.933961 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:27.933941    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-159379b5-serving-cert\""
Jul 24 18:13:27.943491 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:27.943467    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"]
Jul 24 18:13:27.947203 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:27.947175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2df420e1-81f5-492f-91de-a106000d21cd-maas-api-tls\") pod \"maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8\" (UID: \"2df420e1-81f5-492f-91de-a106000d21cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"
Jul 24 18:13:27.947296 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:27.947244    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2df420e1-81f5-492f-91de-a106000d21cd-openshift-service-ca\") pod \"maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8\" (UID: \"2df420e1-81f5-492f-91de-a106000d21cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"
Jul 24 18:13:27.947344 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:27.947319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j78b\" (UniqueName: \"kubernetes.io/projected/2df420e1-81f5-492f-91de-a106000d21cd-kube-api-access-2j78b\") pod \"maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8\" (UID: \"2df420e1-81f5-492f-91de-a106000d21cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"
Jul 24 18:13:28.026437 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.026400    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:28.049009 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.048972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2df420e1-81f5-492f-91de-a106000d21cd-maas-api-tls\") pod \"maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8\" (UID: \"2df420e1-81f5-492f-91de-a106000d21cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"
Jul 24 18:13:28.049326 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.049297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2df420e1-81f5-492f-91de-a106000d21cd-openshift-service-ca\") pod \"maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8\" (UID: \"2df420e1-81f5-492f-91de-a106000d21cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"
Jul 24 18:13:28.049465 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:13:28.049407    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-serving-cert: secret "maas-api-e2e-mt-159379b5-serving-cert" not found
Jul 24 18:13:28.049546 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:13:28.049516    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2df420e1-81f5-492f-91de-a106000d21cd-maas-api-tls podName:2df420e1-81f5-492f-91de-a106000d21cd nodeName:}" failed. No retries permitted until 2026-07-24 18:13:28.549493169 +0000 UTC m=+1831.072851641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2df420e1-81f5-492f-91de-a106000d21cd-maas-api-tls") pod "maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8" (UID: "2df420e1-81f5-492f-91de-a106000d21cd") : secret "maas-api-e2e-mt-159379b5-serving-cert" not found
Jul 24 18:13:28.054512 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.054479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j78b\" (UniqueName: \"kubernetes.io/projected/2df420e1-81f5-492f-91de-a106000d21cd-kube-api-access-2j78b\") pod \"maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8\" (UID: \"2df420e1-81f5-492f-91de-a106000d21cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"
Jul 24 18:13:28.055023 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.055000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2df420e1-81f5-492f-91de-a106000d21cd-openshift-service-ca\") pod \"maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8\" (UID: \"2df420e1-81f5-492f-91de-a106000d21cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"
Jul 24 18:13:28.066475 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.066450    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j78b\" (UniqueName: \"kubernetes.io/projected/2df420e1-81f5-492f-91de-a106000d21cd-kube-api-access-2j78b\") pod \"maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8\" (UID: \"2df420e1-81f5-492f-91de-a106000d21cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"
Jul 24 18:13:28.298626 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.298594    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:28.552132 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.552021    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:28.559727 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.559698    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2df420e1-81f5-492f-91de-a106000d21cd-maas-api-tls\") pod \"maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8\" (UID: \"2df420e1-81f5-492f-91de-a106000d21cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"
Jul 24 18:13:28.562449 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.562420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2df420e1-81f5-492f-91de-a106000d21cd-maas-api-tls\") pod \"maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8\" (UID: \"2df420e1-81f5-492f-91de-a106000d21cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"
Jul 24 18:13:28.647571 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.647539    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:28.754552 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.754517    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn"]
Jul 24 18:13:28.758427 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.758408    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn"
Jul 24 18:13:28.763978 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.763957    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-159379b5\""
Jul 24 18:13:28.764086 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.763957    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-159379b5-dockercfg-k4skw\""
Jul 24 18:13:28.770906 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.770876    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn"]
Jul 24 18:13:28.844041 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.843946    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"
Jul 24 18:13:28.863614 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.863573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22ts5\" (UniqueName: \"kubernetes.io/projected/03ae5052-d87f-4f26-b3c9-ce742a6b0329-kube-api-access-22ts5\") pod \"payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn\" (UID: \"03ae5052-d87f-4f26-b3c9-ce742a6b0329\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn"
Jul 24 18:13:28.863759 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.863643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ae5052-d87f-4f26-b3c9-ce742a6b0329-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn\" (UID: \"03ae5052-d87f-4f26-b3c9-ce742a6b0329\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn"
Jul 24 18:13:28.965106 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.965067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22ts5\" (UniqueName: \"kubernetes.io/projected/03ae5052-d87f-4f26-b3c9-ce742a6b0329-kube-api-access-22ts5\") pod \"payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn\" (UID: \"03ae5052-d87f-4f26-b3c9-ce742a6b0329\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn"
Jul 24 18:13:28.965298 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.965122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ae5052-d87f-4f26-b3c9-ce742a6b0329-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn\" (UID: \"03ae5052-d87f-4f26-b3c9-ce742a6b0329\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn"
Jul 24 18:13:28.965950 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.965924    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ae5052-d87f-4f26-b3c9-ce742a6b0329-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn\" (UID: \"03ae5052-d87f-4f26-b3c9-ce742a6b0329\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn"
Jul 24 18:13:28.975529 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.975502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22ts5\" (UniqueName: \"kubernetes.io/projected/03ae5052-d87f-4f26-b3c9-ce742a6b0329-kube-api-access-22ts5\") pod \"payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn\" (UID: \"03ae5052-d87f-4f26-b3c9-ce742a6b0329\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn"
Jul 24 18:13:28.987967 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.987941    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"]
Jul 24 18:13:28.989958 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:13:28.989935    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df420e1_81f5_492f_91de_a106000d21cd.slice/crio-718e381a389c8cc3106332e9f9dbe30452687a989742a040400a97346820dc36 WatchSource:0}: Error finding container 718e381a389c8cc3106332e9f9dbe30452687a989742a040400a97346820dc36: Status 404 returned error can't find the container with id 718e381a389c8cc3106332e9f9dbe30452687a989742a040400a97346820dc36
Jul 24 18:13:28.991205 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:28.991188    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 18:13:29.070018 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:29.069983    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn"
Jul 24 18:13:29.212552 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:29.212519    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn"]
Jul 24 18:13:29.213668 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:13:29.213634    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ae5052_d87f_4f26_b3c9_ce742a6b0329.slice/crio-04502bc933e20bbc4aaf1c6a692564370e861828e69528bc905abaaab1c0d1df WatchSource:0}: Error finding container 04502bc933e20bbc4aaf1c6a692564370e861828e69528bc905abaaab1c0d1df: Status 404 returned error can't find the container with id 04502bc933e20bbc4aaf1c6a692564370e861828e69528bc905abaaab1c0d1df
Jul 24 18:13:29.852613 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:29.852577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn" event={"ID":"03ae5052-d87f-4f26-b3c9-ce742a6b0329","Type":"ContainerStarted","Data":"8bcb88aac9c4eec3156529b1d963dc3e1308aa500164043700b8b29ae1d4c320"}
Jul 24 18:13:29.853113 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:29.852637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn" event={"ID":"03ae5052-d87f-4f26-b3c9-ce742a6b0329","Type":"ContainerStarted","Data":"04502bc933e20bbc4aaf1c6a692564370e861828e69528bc905abaaab1c0d1df"}
Jul 24 18:13:29.853113 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:29.852676    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn"
Jul 24 18:13:29.854134 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:29.854108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8" event={"ID":"2df420e1-81f5-492f-91de-a106000d21cd","Type":"ContainerStarted","Data":"2a635b3170fc50514a4f1504239aeb177c6a714a1ee0aa423a59b736b535e567"}
Jul 24 18:13:29.854269 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:29.854150    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8" event={"ID":"2df420e1-81f5-492f-91de-a106000d21cd","Type":"ContainerStarted","Data":"718e381a389c8cc3106332e9f9dbe30452687a989742a040400a97346820dc36"}
Jul 24 18:13:29.854269 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:29.854186    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"
Jul 24 18:13:29.869894 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:29.869846    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn" podStartSLOduration=1.869830157 podStartE2EDuration="1.869830157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:13:29.868185415 +0000 UTC m=+1832.391543894" watchObservedRunningTime="2026-07-24 18:13:29.869830157 +0000 UTC m=+1832.393188636"
Jul 24 18:13:29.886313 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:29.886267    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8" podStartSLOduration=2.51237576 podStartE2EDuration="2.886251638s" totalImagesPullingTime="373.875878ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:13:29.88496853 +0000 UTC m=+1832.408327009" watchObservedRunningTime="2026-07-24 18:13:29.886251638 +0000 UTC m=+1832.409610152"
Jul 24 18:13:30.869729 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:30.869697    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:30.970905 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:30.970873    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:32.363428 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:32.363393    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:35.864606 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:35.864577    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"
Jul 24 18:13:40.861709 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:40.861677    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn"
Jul 24 18:13:42.202584 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:13:42.202552    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:14:17.833538 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:17.833501    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"]
Jul 24 18:14:17.834112 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:17.833760    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8" podUID="2df420e1-81f5-492f-91de-a106000d21cd" containerName="maas-api" containerID="cri-o://2a635b3170fc50514a4f1504239aeb177c6a714a1ee0aa423a59b736b535e567" gracePeriod=30
Jul 24 18:14:17.857842 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:17.857805    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:14:17.908810 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:17.908751    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn"]
Jul 24 18:14:17.909426 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:17.909370    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn" podUID="03ae5052-d87f-4f26-b3c9-ce742a6b0329" containerName="payload-pre-processing" containerID="cri-o://8bcb88aac9c4eec3156529b1d963dc3e1308aa500164043700b8b29ae1d4c320" gracePeriod=30
Jul 24 18:14:18.098475 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.098389    2578 generic.go:356] "Generic (PLEG): container finished" podID="03ae5052-d87f-4f26-b3c9-ce742a6b0329" containerID="8bcb88aac9c4eec3156529b1d963dc3e1308aa500164043700b8b29ae1d4c320" exitCode=0
Jul 24 18:14:18.098643 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.098487    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn" event={"ID":"03ae5052-d87f-4f26-b3c9-ce742a6b0329","Type":"ContainerDied","Data":"8bcb88aac9c4eec3156529b1d963dc3e1308aa500164043700b8b29ae1d4c320"}
Jul 24 18:14:18.100444 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.100411    2578 generic.go:356] "Generic (PLEG): container finished" podID="2df420e1-81f5-492f-91de-a106000d21cd" containerID="2a635b3170fc50514a4f1504239aeb177c6a714a1ee0aa423a59b736b535e567" exitCode=0
Jul 24 18:14:18.100613 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.100456    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8" event={"ID":"2df420e1-81f5-492f-91de-a106000d21cd","Type":"ContainerDied","Data":"2a635b3170fc50514a4f1504239aeb177c6a714a1ee0aa423a59b736b535e567"}
Jul 24 18:14:18.160931 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.160909    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"
Jul 24 18:14:18.194879 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.194850    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn"
Jul 24 18:14:18.244637 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.244606    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2df420e1-81f5-492f-91de-a106000d21cd-maas-api-tls\") pod \"2df420e1-81f5-492f-91de-a106000d21cd\" (UID: \"2df420e1-81f5-492f-91de-a106000d21cd\") "
Jul 24 18:14:18.244637 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.244641    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j78b\" (UniqueName: \"kubernetes.io/projected/2df420e1-81f5-492f-91de-a106000d21cd-kube-api-access-2j78b\") pod \"2df420e1-81f5-492f-91de-a106000d21cd\" (UID: \"2df420e1-81f5-492f-91de-a106000d21cd\") "
Jul 24 18:14:18.244876 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.244659    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ae5052-d87f-4f26-b3c9-ce742a6b0329-plugins-config-volume\") pod \"03ae5052-d87f-4f26-b3c9-ce742a6b0329\" (UID: \"03ae5052-d87f-4f26-b3c9-ce742a6b0329\") "
Jul 24 18:14:18.244876 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.244677    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2df420e1-81f5-492f-91de-a106000d21cd-openshift-service-ca\") pod \"2df420e1-81f5-492f-91de-a106000d21cd\" (UID: \"2df420e1-81f5-492f-91de-a106000d21cd\") "
Jul 24 18:14:18.244876 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.244728    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22ts5\" (UniqueName: \"kubernetes.io/projected/03ae5052-d87f-4f26-b3c9-ce742a6b0329-kube-api-access-22ts5\") pod \"03ae5052-d87f-4f26-b3c9-ce742a6b0329\" (UID: \"03ae5052-d87f-4f26-b3c9-ce742a6b0329\") "
Jul 24 18:14:18.245121 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.245093    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df420e1-81f5-492f-91de-a106000d21cd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2df420e1-81f5-492f-91de-a106000d21cd" (UID: "2df420e1-81f5-492f-91de-a106000d21cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:14:18.245121 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.245110    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ae5052-d87f-4f26-b3c9-ce742a6b0329-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "03ae5052-d87f-4f26-b3c9-ce742a6b0329" (UID: "03ae5052-d87f-4f26-b3c9-ce742a6b0329"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:14:18.246882 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.246857    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df420e1-81f5-492f-91de-a106000d21cd-kube-api-access-2j78b" (OuterVolumeSpecName: "kube-api-access-2j78b") pod "2df420e1-81f5-492f-91de-a106000d21cd" (UID: "2df420e1-81f5-492f-91de-a106000d21cd"). InnerVolumeSpecName "kube-api-access-2j78b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:14:18.246981 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.246902    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ae5052-d87f-4f26-b3c9-ce742a6b0329-kube-api-access-22ts5" (OuterVolumeSpecName: "kube-api-access-22ts5") pod "03ae5052-d87f-4f26-b3c9-ce742a6b0329" (UID: "03ae5052-d87f-4f26-b3c9-ce742a6b0329"). InnerVolumeSpecName "kube-api-access-22ts5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:14:18.246981 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.246903    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2df420e1-81f5-492f-91de-a106000d21cd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2df420e1-81f5-492f-91de-a106000d21cd" (UID: "2df420e1-81f5-492f-91de-a106000d21cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:14:18.345952 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.345918    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22ts5\" (UniqueName: \"kubernetes.io/projected/03ae5052-d87f-4f26-b3c9-ce742a6b0329-kube-api-access-22ts5\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:14:18.345952 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.345948    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2df420e1-81f5-492f-91de-a106000d21cd-maas-api-tls\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:14:18.345952 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.345958    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j78b\" (UniqueName: \"kubernetes.io/projected/2df420e1-81f5-492f-91de-a106000d21cd-kube-api-access-2j78b\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:14:18.346182 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.345968    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ae5052-d87f-4f26-b3c9-ce742a6b0329-plugins-config-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:14:18.346182 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:18.345976    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2df420e1-81f5-492f-91de-a106000d21cd-openshift-service-ca\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:14:19.017267 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:19.017224    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:14:19.107472 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:19.107375    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn"
Jul 24 18:14:19.107472 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:19.107363    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn" event={"ID":"03ae5052-d87f-4f26-b3c9-ce742a6b0329","Type":"ContainerDied","Data":"04502bc933e20bbc4aaf1c6a692564370e861828e69528bc905abaaab1c0d1df"}
Jul 24 18:14:19.107472 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:19.107461    2578 scope.go:117] "RemoveContainer" containerID="8bcb88aac9c4eec3156529b1d963dc3e1308aa500164043700b8b29ae1d4c320"
Jul 24 18:14:19.108755 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:19.108739    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"
Jul 24 18:14:19.108834 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:19.108803    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8" event={"ID":"2df420e1-81f5-492f-91de-a106000d21cd","Type":"ContainerDied","Data":"718e381a389c8cc3106332e9f9dbe30452687a989742a040400a97346820dc36"}
Jul 24 18:14:19.116651 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:19.116626    2578 scope.go:117] "RemoveContainer" containerID="2a635b3170fc50514a4f1504239aeb177c6a714a1ee0aa423a59b736b535e567"
Jul 24 18:14:19.134105 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:19.134080    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"]
Jul 24 18:14:19.137888 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:19.137865    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-159379b5-5d95c9fffb-8bkl8"]
Jul 24 18:14:19.148361 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:19.148336    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn"]
Jul 24 18:14:19.152457 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:19.152433    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-159379b5-867bbc79fb-8lnrn"]
Jul 24 18:14:19.967541 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:19.967511    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03ae5052-d87f-4f26-b3c9-ce742a6b0329" path="/var/lib/kubelet/pods/03ae5052-d87f-4f26-b3c9-ce742a6b0329/volumes"
Jul 24 18:14:19.967935 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:19.967921    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2df420e1-81f5-492f-91de-a106000d21cd" path="/var/lib/kubelet/pods/2df420e1-81f5-492f-91de-a106000d21cd/volumes"
Jul 24 18:14:22.469577 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:22.469547    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:14:25.802228 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:25.802194    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:14:30.840261 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:14:30.840227    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:07.509953 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:07.509910    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:08.073954 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:08.073922    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:13.812275 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:13.812239    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:14.023347 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:14.023304    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:14.105665 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:14.105579    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:14.330917 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:14.330871    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:15.471555 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:15.471524    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:15.621999 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:15.621952    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:16.475618 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:16.475583    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:16.584732 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:16.584690    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:16.700835 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:16.700802    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:18.166885 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:18.166854    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:24.808960 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:24.808900    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:24.917799 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:24.917751    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:25.623995 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:25.623955    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:32.088800 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:32.088731    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:32.208589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:32.208553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:32.329054 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:32.329021    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:42.869392 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:42.869351    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:43.211134 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:43.211043    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:48.957621 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:15:48.957585    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:04.240505 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:04.240424    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:10.775750 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:10.775708    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:10.921788 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:10.921744    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:11.155727 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:11.155701    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:11.527182 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:11.527096    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:12.139533 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:12.139496    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:14.767735 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:14.767692    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:14.862731 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:14.862700    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"]
Jul 24 18:16:14.863021 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:14.862854    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:14.863261 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:14.863243    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2df420e1-81f5-492f-91de-a106000d21cd" containerName="maas-api"
Jul 24 18:16:14.863261 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:14.863261    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2df420e1-81f5-492f-91de-a106000d21cd" containerName="maas-api"
Jul 24 18:16:14.863405 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:14.863282    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03ae5052-d87f-4f26-b3c9-ce742a6b0329" containerName="payload-pre-processing"
Jul 24 18:16:14.863405 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:14.863290    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="03ae5052-d87f-4f26-b3c9-ce742a6b0329" containerName="payload-pre-processing"
Jul 24 18:16:14.863405 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:14.863388    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="03ae5052-d87f-4f26-b3c9-ce742a6b0329" containerName="payload-pre-processing"
Jul 24 18:16:14.863405 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:14.863401    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2df420e1-81f5-492f-91de-a106000d21cd" containerName="maas-api"
Jul 24 18:16:14.866743 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:14.866717    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"
Jul 24 18:16:14.870875 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:14.870854    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-04ecb2-serving-cert\""
Jul 24 18:16:14.880096 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:14.880069    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"]
Jul 24 18:16:14.943986 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:14.943949    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:15.022807 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.022706    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-maas-api-tls\") pod \"maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9\" (UID: \"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"
Jul 24 18:16:15.022966 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.022820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9\" (UID: \"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"
Jul 24 18:16:15.022966 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.022854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljmzt\" (UniqueName: \"kubernetes.io/projected/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-kube-api-access-ljmzt\") pod \"maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9\" (UID: \"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"
Jul 24 18:16:15.124231 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.124184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9\" (UID: \"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"
Jul 24 18:16:15.124416 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.124247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljmzt\" (UniqueName: \"kubernetes.io/projected/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-kube-api-access-ljmzt\") pod \"maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9\" (UID: \"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"
Jul 24 18:16:15.124416 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.124313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-maas-api-tls\") pod \"maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9\" (UID: \"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"
Jul 24 18:16:15.124530 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:16:15.124506    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-serving-cert: secret "maas-api-e2e-inf-a-04ecb2-serving-cert" not found
Jul 24 18:16:15.124602 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:16:15.124588    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-maas-api-tls podName:1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a nodeName:}" failed. No retries permitted until 2026-07-24 18:16:15.624566261 +0000 UTC m=+1998.147924734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-maas-api-tls") pod "maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9" (UID: "1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a") : secret "maas-api-e2e-inf-a-04ecb2-serving-cert" not found
Jul 24 18:16:15.124947 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.124928    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9\" (UID: \"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"
Jul 24 18:16:15.142514 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.142478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljmzt\" (UniqueName: \"kubernetes.io/projected/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-kube-api-access-ljmzt\") pod \"maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9\" (UID: \"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"
Jul 24 18:16:15.236765 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.236718    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:15.534326 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.534292    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:15.630754 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.630718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-maas-api-tls\") pod \"maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9\" (UID: \"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"
Jul 24 18:16:15.633172 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.633144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-maas-api-tls\") pod \"maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9\" (UID: \"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"
Jul 24 18:16:15.635737 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.635708    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:15.636332 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.636310    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m"]
Jul 24 18:16:15.639811 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.639789    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m"
Jul 24 18:16:15.643313 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.643295    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-04ecb2-dockercfg-nhv67\""
Jul 24 18:16:15.643521 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.643500    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-04ecb2\""
Jul 24 18:16:15.650632 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.650611    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m"]
Jul 24 18:16:15.778952 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.778914    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"
Jul 24 18:16:15.833072 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.833023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh8rh\" (UniqueName: \"kubernetes.io/projected/17cb3f80-f340-4e87-bb11-5694113960a4-kube-api-access-bh8rh\") pod \"payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m\" (UID: \"17cb3f80-f340-4e87-bb11-5694113960a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m"
Jul 24 18:16:15.833222 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.833104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17cb3f80-f340-4e87-bb11-5694113960a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m\" (UID: \"17cb3f80-f340-4e87-bb11-5694113960a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m"
Jul 24 18:16:15.923708 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.923678    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"]
Jul 24 18:16:15.925948 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:16:15.925923    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1e6fd0_6776_4a9c_b6ff_1c6f41d2c52a.slice/crio-d46f776afa7288f37ba3db475561013614347648e16b0ce7ae3ed1b2ee9e9081 WatchSource:0}: Error finding container d46f776afa7288f37ba3db475561013614347648e16b0ce7ae3ed1b2ee9e9081: Status 404 returned error can't find the container with id d46f776afa7288f37ba3db475561013614347648e16b0ce7ae3ed1b2ee9e9081
Jul 24 18:16:15.933947 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.933921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh8rh\" (UniqueName: \"kubernetes.io/projected/17cb3f80-f340-4e87-bb11-5694113960a4-kube-api-access-bh8rh\") pod \"payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m\" (UID: \"17cb3f80-f340-4e87-bb11-5694113960a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m"
Jul 24 18:16:15.934040 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.933969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17cb3f80-f340-4e87-bb11-5694113960a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m\" (UID: \"17cb3f80-f340-4e87-bb11-5694113960a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m"
Jul 24 18:16:15.934685 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.934667    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17cb3f80-f340-4e87-bb11-5694113960a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m\" (UID: \"17cb3f80-f340-4e87-bb11-5694113960a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m"
Jul 24 18:16:15.943067 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.943044    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh8rh\" (UniqueName: \"kubernetes.io/projected/17cb3f80-f340-4e87-bb11-5694113960a4-kube-api-access-bh8rh\") pod \"payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m\" (UID: \"17cb3f80-f340-4e87-bb11-5694113960a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m"
Jul 24 18:16:15.950926 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:15.950898    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m"
Jul 24 18:16:16.111913 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:16.111803    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m"]
Jul 24 18:16:16.115173 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:16:16.115143    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17cb3f80_f340_4e87_bb11_5694113960a4.slice/crio-4ba4f46763cf13ba6b65588aa341055170194699719ef6d7a8ae8a5c8b2f9e46 WatchSource:0}: Error finding container 4ba4f46763cf13ba6b65588aa341055170194699719ef6d7a8ae8a5c8b2f9e46: Status 404 returned error can't find the container with id 4ba4f46763cf13ba6b65588aa341055170194699719ef6d7a8ae8a5c8b2f9e46
Jul 24 18:16:16.698506 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:16.698384    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9" event={"ID":"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a","Type":"ContainerStarted","Data":"cf7b2078fb00a1febc37f1172bba672019db34f22796ff74236a3fa78fd7f76e"}
Jul 24 18:16:16.698506 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:16.698443    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9" event={"ID":"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a","Type":"ContainerStarted","Data":"d46f776afa7288f37ba3db475561013614347648e16b0ce7ae3ed1b2ee9e9081"}
Jul 24 18:16:16.698506 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:16.698507    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"
Jul 24 18:16:16.700102 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:16.700078    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m" event={"ID":"17cb3f80-f340-4e87-bb11-5694113960a4","Type":"ContainerStarted","Data":"1d94ae7e5187bca1f29980cd82d7fd8ea8648627da03c454c4207ad4031e28b6"}
Jul 24 18:16:16.700253 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:16.700126    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m" event={"ID":"17cb3f80-f340-4e87-bb11-5694113960a4","Type":"ContainerStarted","Data":"4ba4f46763cf13ba6b65588aa341055170194699719ef6d7a8ae8a5c8b2f9e46"}
Jul 24 18:16:16.700253 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:16.700174    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m"
Jul 24 18:16:16.720156 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:16.720107    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9" podStartSLOduration=2.284999767 podStartE2EDuration="2.720088688s" totalImagesPullingTime="435.088921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:16:16.717291965 +0000 UTC m=+1999.240650444" watchObservedRunningTime="2026-07-24 18:16:16.720088688 +0000 UTC m=+1999.243447166"
Jul 24 18:16:16.737075 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:16.737027    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m" podStartSLOduration=1.737012128 podStartE2EDuration="1.737012128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:16:16.735477957 +0000 UTC m=+1999.258836453" watchObservedRunningTime="2026-07-24 18:16:16.737012128 +0000 UTC m=+1999.260370606"
Jul 24 18:16:19.292825 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:19.292759    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:19.631627 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:19.631592    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:19.935452 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:19.935316    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:20.469179 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:20.469125    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:22.713032 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:22.713004    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"
Jul 24 18:16:22.974001 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:22.973856    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:23.079805 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:23.079756    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"]
Jul 24 18:16:23.083483 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:23.083465    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"
Jul 24 18:16:23.086230 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:23.086208    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-dd7b67-serving-cert\""
Jul 24 18:16:23.103585 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:23.103557    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"]
Jul 24 18:16:23.108258 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:23.108230    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac88a61c-595f-490c-bd64-da06dab7c0e2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl\" (UID: \"ac88a61c-595f-490c-bd64-da06dab7c0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"
Jul 24 18:16:23.108395 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:23.108261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cswcv\" (UniqueName: \"kubernetes.io/projected/ac88a61c-595f-490c-bd64-da06dab7c0e2-kube-api-access-cswcv\") pod \"maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl\" (UID: \"ac88a61c-595f-490c-bd64-da06dab7c0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"
Jul 24 18:16:23.108395 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:23.108308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac88a61c-595f-490c-bd64-da06dab7c0e2-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl\" (UID: \"ac88a61c-595f-490c-bd64-da06dab7c0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"
Jul 24 18:16:23.149188 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:23.149157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:23.209448 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:23.209411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac88a61c-595f-490c-bd64-da06dab7c0e2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl\" (UID: \"ac88a61c-595f-490c-bd64-da06dab7c0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"
Jul 24 18:16:23.209649 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:23.209457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cswcv\" (UniqueName: \"kubernetes.io/projected/ac88a61c-595f-490c-bd64-da06dab7c0e2-kube-api-access-cswcv\") pod \"maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl\" (UID: \"ac88a61c-595f-490c-bd64-da06dab7c0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"
Jul 24 18:16:23.209649 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:23.209529    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac88a61c-595f-490c-bd64-da06dab7c0e2-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl\" (UID: \"ac88a61c-595f-490c-bd64-da06dab7c0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"
Jul 24 18:16:23.209649 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:16:23.209637    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-serving-cert: secret "maas-api-e2e-inf-b-dd7b67-serving-cert" not found
Jul 24 18:16:23.209868 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:16:23.209711    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac88a61c-595f-490c-bd64-da06dab7c0e2-maas-api-tls podName:ac88a61c-595f-490c-bd64-da06dab7c0e2 nodeName:}" failed. No retries permitted until 2026-07-24 18:16:23.709691367 +0000 UTC m=+2006.233049830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac88a61c-595f-490c-bd64-da06dab7c0e2-maas-api-tls") pod "maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl" (UID: "ac88a61c-595f-490c-bd64-da06dab7c0e2") : secret "maas-api-e2e-inf-b-dd7b67-serving-cert" not found
Jul 24 18:16:23.210180 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:23.210156    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac88a61c-595f-490c-bd64-da06dab7c0e2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl\" (UID: \"ac88a61c-595f-490c-bd64-da06dab7c0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"
Jul 24 18:16:23.227477 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:23.227410    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cswcv\" (UniqueName: \"kubernetes.io/projected/ac88a61c-595f-490c-bd64-da06dab7c0e2-kube-api-access-cswcv\") pod \"maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl\" (UID: \"ac88a61c-595f-490c-bd64-da06dab7c0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"
Jul 24 18:16:23.263311 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:23.263282    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:23.714556 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:23.714517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac88a61c-595f-490c-bd64-da06dab7c0e2-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl\" (UID: \"ac88a61c-595f-490c-bd64-da06dab7c0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"
Jul 24 18:16:23.717036 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:23.717011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac88a61c-595f-490c-bd64-da06dab7c0e2-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl\" (UID: \"ac88a61c-595f-490c-bd64-da06dab7c0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"
Jul 24 18:16:23.747928 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:23.747894    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:23.977899 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:23.977820    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:23.994947 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:23.994919    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"
Jul 24 18:16:24.140488 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:24.140459    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:24.166098 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:16:24.166061    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac88a61c_595f_490c_bd64_da06dab7c0e2.slice/crio-525b75b9e021258a5f39b8c0fb6b9d403c253ec87082eff82e5052e8bbddf0f4 WatchSource:0}: Error finding container 525b75b9e021258a5f39b8c0fb6b9d403c253ec87082eff82e5052e8bbddf0f4: Status 404 returned error can't find the container with id 525b75b9e021258a5f39b8c0fb6b9d403c253ec87082eff82e5052e8bbddf0f4
Jul 24 18:16:24.169839 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:24.169814    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"]
Jul 24 18:16:24.746075 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:24.746043    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl" event={"ID":"ac88a61c-595f-490c-bd64-da06dab7c0e2","Type":"ContainerStarted","Data":"96b5816cf5844cf89220273464d3574295aaf32b7c24a4c0a261501f74a637a0"}
Jul 24 18:16:24.746456 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:24.746097    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl" event={"ID":"ac88a61c-595f-490c-bd64-da06dab7c0e2","Type":"ContainerStarted","Data":"525b75b9e021258a5f39b8c0fb6b9d403c253ec87082eff82e5052e8bbddf0f4"}
Jul 24 18:16:24.746456 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:24.746156    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"
Jul 24 18:16:24.773913 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:24.773856    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl" podStartSLOduration=1.404252541 podStartE2EDuration="1.773839524s" totalImagesPullingTime="369.586983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:16:24.773094152 +0000 UTC m=+2007.296452631" watchObservedRunningTime="2026-07-24 18:16:24.773839524 +0000 UTC m=+2007.297198002"
Jul 24 18:16:27.362033 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:27.361998    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:27.709925 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:27.709837    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m"
Jul 24 18:16:30.757053 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:30.757022    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"
Jul 24 18:16:40.786150 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:16:40.786116    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:17:06.884427 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:06.884382    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:17:52.594092 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:52.594052    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:17:53.504043 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:53.504012    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:17:53.626215 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:53.626181    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04ecb2-vztjk"]
Jul 24 18:17:53.629897 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:53.629874    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04ecb2-vztjk"
Jul 24 18:17:53.632278 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:53.632255    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgh7n\""
Jul 24 18:17:53.636732 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:53.636708    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04ecb2-vztjk"]
Jul 24 18:17:53.801291 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:53.801261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bde4816-cb67-453b-8d86-a1daf3c7e25a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-04ecb2-vztjk\" (UID: \"8bde4816-cb67-453b-8d86-a1daf3c7e25a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04ecb2-vztjk"
Jul 24 18:17:53.902328 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:53.902279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bde4816-cb67-453b-8d86-a1daf3c7e25a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-04ecb2-vztjk\" (UID: \"8bde4816-cb67-453b-8d86-a1daf3c7e25a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04ecb2-vztjk"
Jul 24 18:17:53.902964 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:53.902944    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bde4816-cb67-453b-8d86-a1daf3c7e25a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-04ecb2-vztjk\" (UID: \"8bde4816-cb67-453b-8d86-a1daf3c7e25a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04ecb2-vztjk"
Jul 24 18:17:53.942354 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:53.942317    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04ecb2-vztjk"
Jul 24 18:17:54.091560 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:54.091479    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04ecb2-vztjk"]
Jul 24 18:17:54.094762 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:17:54.094733    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bde4816_cb67_453b_8d86_a1daf3c7e25a.slice/crio-670a9766708c616c93aa611a4c038002b0222149796ab3e8a3ce5b90d5f79132 WatchSource:0}: Error finding container 670a9766708c616c93aa611a4c038002b0222149796ab3e8a3ce5b90d5f79132: Status 404 returned error can't find the container with id 670a9766708c616c93aa611a4c038002b0222149796ab3e8a3ce5b90d5f79132
Jul 24 18:17:54.188611 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:54.188572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04ecb2-vztjk" event={"ID":"8bde4816-cb67-453b-8d86-a1daf3c7e25a","Type":"ContainerStarted","Data":"670a9766708c616c93aa611a4c038002b0222149796ab3e8a3ce5b90d5f79132"}
Jul 24 18:17:57.207988 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:57.207948    2578 generic.go:356] "Generic (PLEG): container finished" podID="8bde4816-cb67-453b-8d86-a1daf3c7e25a" containerID="5c299e8960512996c23729e58d8b7f056c646a6845105fe1b07013724363c205" exitCode=0
Jul 24 18:17:57.208443 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:57.208033    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04ecb2-vztjk" event={"ID":"8bde4816-cb67-453b-8d86-a1daf3c7e25a","Type":"ContainerDied","Data":"5c299e8960512996c23729e58d8b7f056c646a6845105fe1b07013724363c205"}
Jul 24 18:17:58.350531 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:58.350504    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04ecb2-vztjk"
Jul 24 18:17:58.448046 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:58.448016    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bde4816-cb67-453b-8d86-a1daf3c7e25a-maas-api-service-ca\") pod \"8bde4816-cb67-453b-8d86-a1daf3c7e25a\" (UID: \"8bde4816-cb67-453b-8d86-a1daf3c7e25a\") "
Jul 24 18:17:58.448432 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:58.448411    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bde4816-cb67-453b-8d86-a1daf3c7e25a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8bde4816-cb67-453b-8d86-a1daf3c7e25a" (UID: "8bde4816-cb67-453b-8d86-a1daf3c7e25a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:17:58.549604 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:58.549568    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bde4816-cb67-453b-8d86-a1daf3c7e25a-maas-api-service-ca\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:17:59.218636 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:59.218601    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04ecb2-vztjk" event={"ID":"8bde4816-cb67-453b-8d86-a1daf3c7e25a","Type":"ContainerDied","Data":"670a9766708c616c93aa611a4c038002b0222149796ab3e8a3ce5b90d5f79132"}
Jul 24 18:17:59.218636 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:59.218622    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04ecb2-vztjk"
Jul 24 18:17:59.218886 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:17:59.218657    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="670a9766708c616c93aa611a4c038002b0222149796ab3e8a3ce5b90d5f79132"
Jul 24 18:18:04.023996 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.023960    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"]
Jul 24 18:18:04.024433 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.024234    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9" podUID="1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a" containerName="maas-api" containerID="cri-o://cf7b2078fb00a1febc37f1172bba672019db34f22796ff74236a3fa78fd7f76e" gracePeriod=30
Jul 24 18:18:04.061260 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.061228    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:04.091978 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.091944    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m"]
Jul 24 18:18:04.092385 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.092352    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m" podUID="17cb3f80-f340-4e87-bb11-5694113960a4" containerName="payload-pre-processing" containerID="cri-o://1d94ae7e5187bca1f29980cd82d7fd8ea8648627da03c454c4207ad4031e28b6" gracePeriod=30
Jul 24 18:18:04.160998 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.160970    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:04.251565 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.251527    2578 generic.go:356] "Generic (PLEG): container finished" podID="17cb3f80-f340-4e87-bb11-5694113960a4" containerID="1d94ae7e5187bca1f29980cd82d7fd8ea8648627da03c454c4207ad4031e28b6" exitCode=0
Jul 24 18:18:04.251739 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.251616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m" event={"ID":"17cb3f80-f340-4e87-bb11-5694113960a4","Type":"ContainerDied","Data":"1d94ae7e5187bca1f29980cd82d7fd8ea8648627da03c454c4207ad4031e28b6"}
Jul 24 18:18:04.261705 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.261679    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:04.387801 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.387757    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m"
Jul 24 18:18:04.402747 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.401497    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17cb3f80-f340-4e87-bb11-5694113960a4-plugins-config-volume\") pod \"17cb3f80-f340-4e87-bb11-5694113960a4\" (UID: \"17cb3f80-f340-4e87-bb11-5694113960a4\") "
Jul 24 18:18:04.402747 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.401575    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh8rh\" (UniqueName: \"kubernetes.io/projected/17cb3f80-f340-4e87-bb11-5694113960a4-kube-api-access-bh8rh\") pod \"17cb3f80-f340-4e87-bb11-5694113960a4\" (UID: \"17cb3f80-f340-4e87-bb11-5694113960a4\") "
Jul 24 18:18:04.402997 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.402736    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17cb3f80-f340-4e87-bb11-5694113960a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "17cb3f80-f340-4e87-bb11-5694113960a4" (UID: "17cb3f80-f340-4e87-bb11-5694113960a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:18:04.407678 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.407566    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17cb3f80-f340-4e87-bb11-5694113960a4-kube-api-access-bh8rh" (OuterVolumeSpecName: "kube-api-access-bh8rh") pod "17cb3f80-f340-4e87-bb11-5694113960a4" (UID: "17cb3f80-f340-4e87-bb11-5694113960a4"). InnerVolumeSpecName "kube-api-access-bh8rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:18:04.474391 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.474368    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"
Jul 24 18:18:04.502757 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.502724    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljmzt\" (UniqueName: \"kubernetes.io/projected/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-kube-api-access-ljmzt\") pod \"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a\" (UID: \"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a\") "
Jul 24 18:18:04.502940 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.502807    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-openshift-service-ca\") pod \"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a\" (UID: \"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a\") "
Jul 24 18:18:04.502940 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.502884    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-maas-api-tls\") pod \"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a\" (UID: \"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a\") "
Jul 24 18:18:04.503211 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.503186    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh8rh\" (UniqueName: \"kubernetes.io/projected/17cb3f80-f340-4e87-bb11-5694113960a4-kube-api-access-bh8rh\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:18:04.503281 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.503215    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17cb3f80-f340-4e87-bb11-5694113960a4-plugins-config-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:18:04.503457 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.503429    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a" (UID: "1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:18:04.504918 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.504886    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-kube-api-access-ljmzt" (OuterVolumeSpecName: "kube-api-access-ljmzt") pod "1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a" (UID: "1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a"). InnerVolumeSpecName "kube-api-access-ljmzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:18:04.505123 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.505095    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a" (UID: "1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:18:04.604327 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.604243    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-maas-api-tls\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:18:04.604327 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.604272    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljmzt\" (UniqueName: \"kubernetes.io/projected/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-kube-api-access-ljmzt\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:18:04.604327 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:04.604283    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a-openshift-service-ca\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:18:05.258551 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:05.258514    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m"
Jul 24 18:18:05.259026 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:05.258547    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m" event={"ID":"17cb3f80-f340-4e87-bb11-5694113960a4","Type":"ContainerDied","Data":"4ba4f46763cf13ba6b65588aa341055170194699719ef6d7a8ae8a5c8b2f9e46"}
Jul 24 18:18:05.259026 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:05.258605    2578 scope.go:117] "RemoveContainer" containerID="1d94ae7e5187bca1f29980cd82d7fd8ea8648627da03c454c4207ad4031e28b6"
Jul 24 18:18:05.259974 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:05.259934    2578 generic.go:356] "Generic (PLEG): container finished" podID="1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a" containerID="cf7b2078fb00a1febc37f1172bba672019db34f22796ff74236a3fa78fd7f76e" exitCode=0
Jul 24 18:18:05.260032 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:05.259972    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9" event={"ID":"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a","Type":"ContainerDied","Data":"cf7b2078fb00a1febc37f1172bba672019db34f22796ff74236a3fa78fd7f76e"}
Jul 24 18:18:05.260032 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:05.259980    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"
Jul 24 18:18:05.260032 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:05.260009    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9" event={"ID":"1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a","Type":"ContainerDied","Data":"d46f776afa7288f37ba3db475561013614347648e16b0ce7ae3ed1b2ee9e9081"}
Jul 24 18:18:05.268623 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:05.268606    2578 scope.go:117] "RemoveContainer" containerID="cf7b2078fb00a1febc37f1172bba672019db34f22796ff74236a3fa78fd7f76e"
Jul 24 18:18:05.276720 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:05.276703    2578 scope.go:117] "RemoveContainer" containerID="cf7b2078fb00a1febc37f1172bba672019db34f22796ff74236a3fa78fd7f76e"
Jul 24 18:18:05.277020 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:18:05.276987    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf7b2078fb00a1febc37f1172bba672019db34f22796ff74236a3fa78fd7f76e\": container with ID starting with cf7b2078fb00a1febc37f1172bba672019db34f22796ff74236a3fa78fd7f76e not found: ID does not exist" containerID="cf7b2078fb00a1febc37f1172bba672019db34f22796ff74236a3fa78fd7f76e"
Jul 24 18:18:05.277116 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:05.277022    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7b2078fb00a1febc37f1172bba672019db34f22796ff74236a3fa78fd7f76e"} err="failed to get container status \"cf7b2078fb00a1febc37f1172bba672019db34f22796ff74236a3fa78fd7f76e\": rpc error: code = NotFound desc = could not find container \"cf7b2078fb00a1febc37f1172bba672019db34f22796ff74236a3fa78fd7f76e\": container with ID starting with cf7b2078fb00a1febc37f1172bba672019db34f22796ff74236a3fa78fd7f76e not found: ID does not exist"
Jul 24 18:18:05.299138 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:05.299105    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"]
Jul 24 18:18:05.308237 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:05.308207    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04ecb2-99df774cc-hrzg9"]
Jul 24 18:18:05.320164 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:05.320125    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m"]
Jul 24 18:18:05.324972 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:05.324948    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04ecb2-68b7894496-2sv5m"]
Jul 24 18:18:05.968859 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:05.968824    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17cb3f80-f340-4e87-bb11-5694113960a4" path="/var/lib/kubelet/pods/17cb3f80-f340-4e87-bb11-5694113960a4/volumes"
Jul 24 18:18:05.969253 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:05.969239    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a" path="/var/lib/kubelet/pods/1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a/volumes"
Jul 24 18:18:08.700630 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:08.700595    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:08.713742 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:08.713710    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04ecb2-vztjk"]
Jul 24 18:18:08.716431 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:08.716399    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04ecb2-vztjk"]
Jul 24 18:18:09.106723 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:09.106691    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:09.968247 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:09.968212    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bde4816-cb67-453b-8d86-a1daf3c7e25a" path="/var/lib/kubelet/pods/8bde4816-cb67-453b-8d86-a1daf3c7e25a/volumes"
Jul 24 18:18:14.991522 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:14.991484    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:15.057077 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.057042    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:15.310289 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.310254    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:15.540659 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.540625    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:15.661858 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.661745    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd7b67-j8jzc"]
Jul 24 18:18:15.662272 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.662253    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bde4816-cb67-453b-8d86-a1daf3c7e25a" containerName="revoke-keys"
Jul 24 18:18:15.662369 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.662275    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bde4816-cb67-453b-8d86-a1daf3c7e25a" containerName="revoke-keys"
Jul 24 18:18:15.662369 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.662328    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17cb3f80-f340-4e87-bb11-5694113960a4" containerName="payload-pre-processing"
Jul 24 18:18:15.662369 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.662337    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="17cb3f80-f340-4e87-bb11-5694113960a4" containerName="payload-pre-processing"
Jul 24 18:18:15.662369 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.662350    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a" containerName="maas-api"
Jul 24 18:18:15.662369 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.662359    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a" containerName="maas-api"
Jul 24 18:18:15.662610 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.662452    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bde4816-cb67-453b-8d86-a1daf3c7e25a" containerName="revoke-keys"
Jul 24 18:18:15.662610 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.662467    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c1e6fd0-6776-4a9c-b6ff-1c6f41d2c52a" containerName="maas-api"
Jul 24 18:18:15.662610 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.662477    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="17cb3f80-f340-4e87-bb11-5694113960a4" containerName="payload-pre-processing"
Jul 24 18:18:15.667024 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.667002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd7b67-j8jzc"
Jul 24 18:18:15.669978 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.669954    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgh7n\""
Jul 24 18:18:15.672835 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.672756    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd7b67-j8jzc"]
Jul 24 18:18:15.710888 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.710853    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d7932d-45f4-42bf-9a63-c75bed155061-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd7b67-j8jzc\" (UID: \"39d7932d-45f4-42bf-9a63-c75bed155061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd7b67-j8jzc"
Jul 24 18:18:15.811493 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.811461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d7932d-45f4-42bf-9a63-c75bed155061-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd7b67-j8jzc\" (UID: \"39d7932d-45f4-42bf-9a63-c75bed155061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd7b67-j8jzc"
Jul 24 18:18:15.812143 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.812121    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d7932d-45f4-42bf-9a63-c75bed155061-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd7b67-j8jzc\" (UID: \"39d7932d-45f4-42bf-9a63-c75bed155061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd7b67-j8jzc"
Jul 24 18:18:15.980495 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:15.980402    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd7b67-j8jzc"
Jul 24 18:18:16.142647 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:16.142614    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd7b67-j8jzc"]
Jul 24 18:18:16.146331 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:18:16.146304    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d7932d_45f4_42bf_9a63_c75bed155061.slice/crio-11cdd17e8fc4a94b0c0028b07a3d28835f41710f28724362a158031abbd9b961 WatchSource:0}: Error finding container 11cdd17e8fc4a94b0c0028b07a3d28835f41710f28724362a158031abbd9b961: Status 404 returned error can't find the container with id 11cdd17e8fc4a94b0c0028b07a3d28835f41710f28724362a158031abbd9b961
Jul 24 18:18:16.319281 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:16.319244    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd7b67-j8jzc" event={"ID":"39d7932d-45f4-42bf-9a63-c75bed155061","Type":"ContainerStarted","Data":"11cdd17e8fc4a94b0c0028b07a3d28835f41710f28724362a158031abbd9b961"}
Jul 24 18:18:17.325790 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:17.325746    2578 generic.go:356] "Generic (PLEG): container finished" podID="39d7932d-45f4-42bf-9a63-c75bed155061" containerID="fa1855168476f22fc4e6a39a192423c20fe3d21f3fb22ba509772399d594f935" exitCode=0
Jul 24 18:18:17.326181 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:17.325812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd7b67-j8jzc" event={"ID":"39d7932d-45f4-42bf-9a63-c75bed155061","Type":"ContainerDied","Data":"fa1855168476f22fc4e6a39a192423c20fe3d21f3fb22ba509772399d594f935"}
Jul 24 18:18:18.471789 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:18.471755    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd7b67-j8jzc"
Jul 24 18:18:18.540503 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:18.540468    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d7932d-45f4-42bf-9a63-c75bed155061-maas-api-service-ca\") pod \"39d7932d-45f4-42bf-9a63-c75bed155061\" (UID: \"39d7932d-45f4-42bf-9a63-c75bed155061\") "
Jul 24 18:18:18.540837 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:18.540813    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d7932d-45f4-42bf-9a63-c75bed155061-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "39d7932d-45f4-42bf-9a63-c75bed155061" (UID: "39d7932d-45f4-42bf-9a63-c75bed155061"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:18:18.641488 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:18.641391    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d7932d-45f4-42bf-9a63-c75bed155061-maas-api-service-ca\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:18:19.336770 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:19.336738    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd7b67-j8jzc"
Jul 24 18:18:19.336770 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:19.336768    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd7b67-j8jzc" event={"ID":"39d7932d-45f4-42bf-9a63-c75bed155061","Type":"ContainerDied","Data":"11cdd17e8fc4a94b0c0028b07a3d28835f41710f28724362a158031abbd9b961"}
Jul 24 18:18:19.337006 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:19.336833    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11cdd17e8fc4a94b0c0028b07a3d28835f41710f28724362a158031abbd9b961"
Jul 24 18:18:26.074135 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:26.074092    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"]
Jul 24 18:18:26.075843 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:26.074383    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl" podUID="ac88a61c-595f-490c-bd64-da06dab7c0e2" containerName="maas-api" containerID="cri-o://96b5816cf5844cf89220273464d3574295aaf32b7c24a4c0a261501f74a637a0" gracePeriod=30
Jul 24 18:18:26.108506 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:26.108467    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:26.239740 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:26.239710    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:26.395113 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:26.395081    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:26.551314 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:26.551290    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"
Jul 24 18:18:26.629521 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:26.629443    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac88a61c-595f-490c-bd64-da06dab7c0e2-openshift-service-ca\") pod \"ac88a61c-595f-490c-bd64-da06dab7c0e2\" (UID: \"ac88a61c-595f-490c-bd64-da06dab7c0e2\") "
Jul 24 18:18:26.629521 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:26.629494    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cswcv\" (UniqueName: \"kubernetes.io/projected/ac88a61c-595f-490c-bd64-da06dab7c0e2-kube-api-access-cswcv\") pod \"ac88a61c-595f-490c-bd64-da06dab7c0e2\" (UID: \"ac88a61c-595f-490c-bd64-da06dab7c0e2\") "
Jul 24 18:18:26.629708 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:26.629538    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac88a61c-595f-490c-bd64-da06dab7c0e2-maas-api-tls\") pod \"ac88a61c-595f-490c-bd64-da06dab7c0e2\" (UID: \"ac88a61c-595f-490c-bd64-da06dab7c0e2\") "
Jul 24 18:18:26.629818 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:26.629795    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac88a61c-595f-490c-bd64-da06dab7c0e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ac88a61c-595f-490c-bd64-da06dab7c0e2" (UID: "ac88a61c-595f-490c-bd64-da06dab7c0e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:18:26.631584 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:26.631556    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac88a61c-595f-490c-bd64-da06dab7c0e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ac88a61c-595f-490c-bd64-da06dab7c0e2" (UID: "ac88a61c-595f-490c-bd64-da06dab7c0e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:18:26.631692 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:26.631611    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac88a61c-595f-490c-bd64-da06dab7c0e2-kube-api-access-cswcv" (OuterVolumeSpecName: "kube-api-access-cswcv") pod "ac88a61c-595f-490c-bd64-da06dab7c0e2" (UID: "ac88a61c-595f-490c-bd64-da06dab7c0e2"). InnerVolumeSpecName "kube-api-access-cswcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:18:26.731125 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:26.731085    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac88a61c-595f-490c-bd64-da06dab7c0e2-openshift-service-ca\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:18:26.731125 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:26.731119    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cswcv\" (UniqueName: \"kubernetes.io/projected/ac88a61c-595f-490c-bd64-da06dab7c0e2-kube-api-access-cswcv\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:18:26.731125 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:26.731129    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac88a61c-595f-490c-bd64-da06dab7c0e2-maas-api-tls\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:18:27.377618 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:27.377586    2578 generic.go:356] "Generic (PLEG): container finished" podID="ac88a61c-595f-490c-bd64-da06dab7c0e2" containerID="96b5816cf5844cf89220273464d3574295aaf32b7c24a4c0a261501f74a637a0" exitCode=0
Jul 24 18:18:27.378088 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:27.377632    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"
Jul 24 18:18:27.378088 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:27.377684    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl" event={"ID":"ac88a61c-595f-490c-bd64-da06dab7c0e2","Type":"ContainerDied","Data":"96b5816cf5844cf89220273464d3574295aaf32b7c24a4c0a261501f74a637a0"}
Jul 24 18:18:27.378088 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:27.377732    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl" event={"ID":"ac88a61c-595f-490c-bd64-da06dab7c0e2","Type":"ContainerDied","Data":"525b75b9e021258a5f39b8c0fb6b9d403c253ec87082eff82e5052e8bbddf0f4"}
Jul 24 18:18:27.378088 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:27.377755    2578 scope.go:117] "RemoveContainer" containerID="96b5816cf5844cf89220273464d3574295aaf32b7c24a4c0a261501f74a637a0"
Jul 24 18:18:27.386896 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:27.386878    2578 scope.go:117] "RemoveContainer" containerID="96b5816cf5844cf89220273464d3574295aaf32b7c24a4c0a261501f74a637a0"
Jul 24 18:18:27.387156 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:18:27.387138    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b5816cf5844cf89220273464d3574295aaf32b7c24a4c0a261501f74a637a0\": container with ID starting with 96b5816cf5844cf89220273464d3574295aaf32b7c24a4c0a261501f74a637a0 not found: ID does not exist" containerID="96b5816cf5844cf89220273464d3574295aaf32b7c24a4c0a261501f74a637a0"
Jul 24 18:18:27.387224 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:27.387165    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b5816cf5844cf89220273464d3574295aaf32b7c24a4c0a261501f74a637a0"} err="failed to get container status \"96b5816cf5844cf89220273464d3574295aaf32b7c24a4c0a261501f74a637a0\": rpc error: code = NotFound desc = could not find container \"96b5816cf5844cf89220273464d3574295aaf32b7c24a4c0a261501f74a637a0\": container with ID starting with 96b5816cf5844cf89220273464d3574295aaf32b7c24a4c0a261501f74a637a0 not found: ID does not exist"
Jul 24 18:18:27.401386 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:27.401362    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"]
Jul 24 18:18:27.406550 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:27.406525    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd7b67-7d56ccfd4-2czrl"]
Jul 24 18:18:27.967493 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:27.967459    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac88a61c-595f-490c-bd64-da06dab7c0e2" path="/var/lib/kubelet/pods/ac88a61c-595f-490c-bd64-da06dab7c0e2/volumes"
Jul 24 18:18:30.775379 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:30.775341    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd7b67-j8jzc"]
Jul 24 18:18:30.777317 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:30.777280    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:30.785641 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:30.785618    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd7b67-j8jzc"]
Jul 24 18:18:31.182712 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:31.182676    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:31.968299 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:31.968257    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39d7932d-45f4-42bf-9a63-c75bed155061" path="/var/lib/kubelet/pods/39d7932d-45f4-42bf-9a63-c75bed155061/volumes"
Jul 24 18:18:37.031526 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:37.031493    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:37.133439 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:37.133406    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:37.308360 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:37.308327    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:37.553003 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:37.552964    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:37.911178 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:37.911137    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:38.250735 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:38.250637    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:38.676551 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:38.676517    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:41.791166 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:41.791117    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:41.933962 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:41.933919    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"]
Jul 24 18:18:41.934380 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:41.934364    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac88a61c-595f-490c-bd64-da06dab7c0e2" containerName="maas-api"
Jul 24 18:18:41.934431 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:41.934384    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac88a61c-595f-490c-bd64-da06dab7c0e2" containerName="maas-api"
Jul 24 18:18:41.934431 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:41.934393    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d7932d-45f4-42bf-9a63-c75bed155061" containerName="revoke-keys"
Jul 24 18:18:41.934431 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:41.934398    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d7932d-45f4-42bf-9a63-c75bed155061" containerName="revoke-keys"
Jul 24 18:18:41.934527 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:41.934466    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac88a61c-595f-490c-bd64-da06dab7c0e2" containerName="maas-api"
Jul 24 18:18:41.934527 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:41.934476    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="39d7932d-45f4-42bf-9a63-c75bed155061" containerName="revoke-keys"
Jul 24 18:18:41.936765 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:41.936748    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"
Jul 24 18:18:41.940404 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:41.940377    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ab4b1a-serving-cert\""
Jul 24 18:18:41.967411 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:41.967376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-openshift-service-ca\") pod \"maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd\" (UID: \"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"
Jul 24 18:18:41.967603 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:41.967425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcbzq\" (UniqueName: \"kubernetes.io/projected/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-kube-api-access-rcbzq\") pod \"maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd\" (UID: \"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"
Jul 24 18:18:41.967603 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:41.967587    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-maas-api-tls\") pod \"maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd\" (UID: \"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"
Jul 24 18:18:41.979004 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:41.978978    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"]
Jul 24 18:18:42.025823 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.025762    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:42.068251 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.068209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcbzq\" (UniqueName: \"kubernetes.io/projected/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-kube-api-access-rcbzq\") pod \"maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd\" (UID: \"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"
Jul 24 18:18:42.068476 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.068358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-maas-api-tls\") pod \"maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd\" (UID: \"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"
Jul 24 18:18:42.068476 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.068388    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-openshift-service-ca\") pod \"maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd\" (UID: \"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"
Jul 24 18:18:42.068587 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:18:42.068540    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-serving-cert: secret "maas-api-e2e-api-a-ab4b1a-serving-cert" not found
Jul 24 18:18:42.068647 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:18:42.068633    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-maas-api-tls podName:1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257 nodeName:}" failed. No retries permitted until 2026-07-24 18:18:42.568608128 +0000 UTC m=+2145.091966608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-maas-api-tls") pod "maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd" (UID: "1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257") : secret "maas-api-e2e-api-a-ab4b1a-serving-cert" not found
Jul 24 18:18:42.069058 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.069040    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-openshift-service-ca\") pod \"maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd\" (UID: \"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"
Jul 24 18:18:42.088347 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.088305    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcbzq\" (UniqueName: \"kubernetes.io/projected/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-kube-api-access-rcbzq\") pod \"maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd\" (UID: \"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"
Jul 24 18:18:42.136148 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.136114    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:42.389288 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.389198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:42.573411 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.573374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-maas-api-tls\") pod \"maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd\" (UID: \"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"
Jul 24 18:18:42.576121 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.576096    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-maas-api-tls\") pod \"maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd\" (UID: \"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"
Jul 24 18:18:42.698491 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.698408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:42.798129 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.798098    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:42.846610 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.846573    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"
Jul 24 18:18:42.880843 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.880803    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm"]
Jul 24 18:18:42.883406 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.883384    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm"
Jul 24 18:18:42.886341 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.886317    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ab4b1a\""
Jul 24 18:18:42.886341 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.886334    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ab4b1a-dockercfg-xwz5n\""
Jul 24 18:18:42.894756 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.894725    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm"]
Jul 24 18:18:42.977881 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.977776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16bef317-92a6-426b-962a-723c3699d30d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm\" (UID: \"16bef317-92a6-426b-962a-723c3699d30d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm"
Jul 24 18:18:42.977881 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:42.977851    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th8hc\" (UniqueName: \"kubernetes.io/projected/16bef317-92a6-426b-962a-723c3699d30d-kube-api-access-th8hc\") pod \"payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm\" (UID: \"16bef317-92a6-426b-962a-723c3699d30d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm"
Jul 24 18:18:43.008096 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:43.007901    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"]
Jul 24 18:18:43.010682 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:18:43.010652    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7f0f0e_8cb1_4fbc_861d_aa4e6e6e8257.slice/crio-5f35254ae76fb152ceaaef260505b8fce69a36da31c391366cd128d1f8a62f67 WatchSource:0}: Error finding container 5f35254ae76fb152ceaaef260505b8fce69a36da31c391366cd128d1f8a62f67: Status 404 returned error can't find the container with id 5f35254ae76fb152ceaaef260505b8fce69a36da31c391366cd128d1f8a62f67
Jul 24 18:18:43.011969 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:43.011953    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 18:18:43.080326 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:43.080212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16bef317-92a6-426b-962a-723c3699d30d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm\" (UID: \"16bef317-92a6-426b-962a-723c3699d30d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm"
Jul 24 18:18:43.080516 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:43.080400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th8hc\" (UniqueName: \"kubernetes.io/projected/16bef317-92a6-426b-962a-723c3699d30d-kube-api-access-th8hc\") pod \"payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm\" (UID: \"16bef317-92a6-426b-962a-723c3699d30d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm"
Jul 24 18:18:43.080964 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:43.080942    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16bef317-92a6-426b-962a-723c3699d30d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm\" (UID: \"16bef317-92a6-426b-962a-723c3699d30d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm"
Jul 24 18:18:43.106475 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:43.106433    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th8hc\" (UniqueName: \"kubernetes.io/projected/16bef317-92a6-426b-962a-723c3699d30d-kube-api-access-th8hc\") pod \"payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm\" (UID: \"16bef317-92a6-426b-962a-723c3699d30d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm"
Jul 24 18:18:43.196368 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:43.196319    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm"
Jul 24 18:18:43.336966 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:43.336937    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm"]
Jul 24 18:18:43.338889 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:18:43.338853    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16bef317_92a6_426b_962a_723c3699d30d.slice/crio-a43c1d5a5f4dcb2cf9eb21925949aeadbdedc18f7720bd1041b0f61aaf52301e WatchSource:0}: Error finding container a43c1d5a5f4dcb2cf9eb21925949aeadbdedc18f7720bd1041b0f61aaf52301e: Status 404 returned error can't find the container with id a43c1d5a5f4dcb2cf9eb21925949aeadbdedc18f7720bd1041b0f61aaf52301e
Jul 24 18:18:43.460047 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:43.460012    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd" event={"ID":"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257","Type":"ContainerStarted","Data":"5f35254ae76fb152ceaaef260505b8fce69a36da31c391366cd128d1f8a62f67"}
Jul 24 18:18:43.461638 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:43.461607    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm" event={"ID":"16bef317-92a6-426b-962a-723c3699d30d","Type":"ContainerStarted","Data":"af1f6928e9bf4fc439cb42ddf38ede16205debf9704a1dfbae0329a1d03505b3"}
Jul 24 18:18:43.461809 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:43.461661    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm" event={"ID":"16bef317-92a6-426b-962a-723c3699d30d","Type":"ContainerStarted","Data":"a43c1d5a5f4dcb2cf9eb21925949aeadbdedc18f7720bd1041b0f61aaf52301e"}
Jul 24 18:18:43.461809 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:43.461710    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm"
Jul 24 18:18:43.481478 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:43.481437    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm" podStartSLOduration=1.481422333 podStartE2EDuration="1.481422333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:18:43.479987348 +0000 UTC m=+2146.003345827" watchObservedRunningTime="2026-07-24 18:18:43.481422333 +0000 UTC m=+2146.004780811"
Jul 24 18:18:44.468710 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:44.468673    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd" event={"ID":"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257","Type":"ContainerStarted","Data":"7e5584ee7d0dae0b250d481da9df0b7dfb39cf103fb9272d0ab841e0c2570131"}
Jul 24 18:18:44.487319 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:44.487269    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd" podStartSLOduration=3.02288948 podStartE2EDuration="3.48725025s" totalImagesPullingTime="464.36077ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:18:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:18:44.486883208 +0000 UTC m=+2147.010241690" watchObservedRunningTime="2026-07-24 18:18:44.48725025 +0000 UTC m=+2147.010608728"
Jul 24 18:18:45.474306 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:45.474272    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"
Jul 24 18:18:46.174052 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:46.174019    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:46.661796 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:46.661755    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:46.997475 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:46.997340    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:47.097044 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:47.097006    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:49.522446 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:49.522342    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:49.616365 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:49.616318    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:49.785527 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:49.785446    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"]
Jul 24 18:18:49.792528 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:49.792494    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"
Jul 24 18:18:49.796424 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:49.796395    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ac2de9-serving-cert\""
Jul 24 18:18:49.806412 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:49.806382    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"]
Jul 24 18:18:49.838324 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:49.838285    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:49.847581 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:49.847552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a04b486-2a0f-4290-b591-a3f239470ba9-maas-api-tls\") pod \"maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt\" (UID: \"3a04b486-2a0f-4290-b591-a3f239470ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"
Jul 24 18:18:49.847706 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:49.847588    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bh87\" (UniqueName: \"kubernetes.io/projected/3a04b486-2a0f-4290-b591-a3f239470ba9-kube-api-access-2bh87\") pod \"maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt\" (UID: \"3a04b486-2a0f-4290-b591-a3f239470ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"
Jul 24 18:18:49.847706 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:49.847667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a04b486-2a0f-4290-b591-a3f239470ba9-openshift-service-ca\") pod \"maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt\" (UID: \"3a04b486-2a0f-4290-b591-a3f239470ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"
Jul 24 18:18:49.948927 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:49.948897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a04b486-2a0f-4290-b591-a3f239470ba9-maas-api-tls\") pod \"maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt\" (UID: \"3a04b486-2a0f-4290-b591-a3f239470ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"
Jul 24 18:18:49.949116 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:49.948937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bh87\" (UniqueName: \"kubernetes.io/projected/3a04b486-2a0f-4290-b591-a3f239470ba9-kube-api-access-2bh87\") pod \"maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt\" (UID: \"3a04b486-2a0f-4290-b591-a3f239470ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"
Jul 24 18:18:49.949116 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:49.949018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a04b486-2a0f-4290-b591-a3f239470ba9-openshift-service-ca\") pod \"maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt\" (UID: \"3a04b486-2a0f-4290-b591-a3f239470ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"
Jul 24 18:18:49.949116 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:18:49.949060    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-serving-cert: secret "maas-api-e2e-api-b-ac2de9-serving-cert" not found
Jul 24 18:18:49.949292 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:18:49.949151    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a04b486-2a0f-4290-b591-a3f239470ba9-maas-api-tls podName:3a04b486-2a0f-4290-b591-a3f239470ba9 nodeName:}" failed. No retries permitted until 2026-07-24 18:18:50.449130416 +0000 UTC m=+2152.972488872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a04b486-2a0f-4290-b591-a3f239470ba9-maas-api-tls") pod "maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt" (UID: "3a04b486-2a0f-4290-b591-a3f239470ba9") : secret "maas-api-e2e-api-b-ac2de9-serving-cert" not found
Jul 24 18:18:49.949830 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:49.949807    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a04b486-2a0f-4290-b591-a3f239470ba9-openshift-service-ca\") pod \"maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt\" (UID: \"3a04b486-2a0f-4290-b591-a3f239470ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"
Jul 24 18:18:49.958012 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:49.957992    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bh87\" (UniqueName: \"kubernetes.io/projected/3a04b486-2a0f-4290-b591-a3f239470ba9-kube-api-access-2bh87\") pod \"maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt\" (UID: \"3a04b486-2a0f-4290-b591-a3f239470ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"
Jul 24 18:18:50.109855 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.109822    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:50.359945 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.359853    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:50.446009 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.445968    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:50.454388 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.454330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a04b486-2a0f-4290-b591-a3f239470ba9-maas-api-tls\") pod \"maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt\" (UID: \"3a04b486-2a0f-4290-b591-a3f239470ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"
Jul 24 18:18:50.457023 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.456998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a04b486-2a0f-4290-b591-a3f239470ba9-maas-api-tls\") pod \"maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt\" (UID: \"3a04b486-2a0f-4290-b591-a3f239470ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"
Jul 24 18:18:50.462650 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.462604    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx"]
Jul 24 18:18:50.466021 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.466002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx"
Jul 24 18:18:50.468556 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.468532    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ac2de9\""
Jul 24 18:18:50.468935 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.468915    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ac2de9-dockercfg-26dpc\""
Jul 24 18:18:50.480204 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.480180    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx"]
Jul 24 18:18:50.555874 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.555830    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jxbn\" (UniqueName: \"kubernetes.io/projected/225bb0ec-6f53-4a7c-9412-2c32a3fc1e37-kube-api-access-2jxbn\") pod \"payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx\" (UID: \"225bb0ec-6f53-4a7c-9412-2c32a3fc1e37\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx"
Jul 24 18:18:50.556305 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.555969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225bb0ec-6f53-4a7c-9412-2c32a3fc1e37-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx\" (UID: \"225bb0ec-6f53-4a7c-9412-2c32a3fc1e37\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx"
Jul 24 18:18:50.657077 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.656978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225bb0ec-6f53-4a7c-9412-2c32a3fc1e37-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx\" (UID: \"225bb0ec-6f53-4a7c-9412-2c32a3fc1e37\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx"
Jul 24 18:18:50.657077 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.657075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jxbn\" (UniqueName: \"kubernetes.io/projected/225bb0ec-6f53-4a7c-9412-2c32a3fc1e37-kube-api-access-2jxbn\") pod \"payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx\" (UID: \"225bb0ec-6f53-4a7c-9412-2c32a3fc1e37\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx"
Jul 24 18:18:50.657716 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.657693    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225bb0ec-6f53-4a7c-9412-2c32a3fc1e37-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx\" (UID: \"225bb0ec-6f53-4a7c-9412-2c32a3fc1e37\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx"
Jul 24 18:18:50.666614 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.666578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jxbn\" (UniqueName: \"kubernetes.io/projected/225bb0ec-6f53-4a7c-9412-2c32a3fc1e37-kube-api-access-2jxbn\") pod \"payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx\" (UID: \"225bb0ec-6f53-4a7c-9412-2c32a3fc1e37\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx"
Jul 24 18:18:50.706456 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.706419    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"
Jul 24 18:18:50.778854 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.778824    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx"
Jul 24 18:18:50.859249 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.859223    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"]
Jul 24 18:18:50.860726 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:18:50.860698    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a04b486_2a0f_4290_b591_a3f239470ba9.slice/crio-9855090a473599dcd2faa8d0d70634d3a6127749d4a0bc4253a55821e5b6fe77 WatchSource:0}: Error finding container 9855090a473599dcd2faa8d0d70634d3a6127749d4a0bc4253a55821e5b6fe77: Status 404 returned error can't find the container with id 9855090a473599dcd2faa8d0d70634d3a6127749d4a0bc4253a55821e5b6fe77
Jul 24 18:18:50.935540 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:50.935516    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx"]
Jul 24 18:18:50.937338 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:18:50.937307    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225bb0ec_6f53_4a7c_9412_2c32a3fc1e37.slice/crio-9f1f2ef8d20254f86f43cc58328c9f327120fa0368ab4bc30a68c5d7a4fb20b9 WatchSource:0}: Error finding container 9f1f2ef8d20254f86f43cc58328c9f327120fa0368ab4bc30a68c5d7a4fb20b9: Status 404 returned error can't find the container with id 9f1f2ef8d20254f86f43cc58328c9f327120fa0368ab4bc30a68c5d7a4fb20b9
Jul 24 18:18:51.484775 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:51.484745    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"
Jul 24 18:18:51.510248 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:51.510190    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt" event={"ID":"3a04b486-2a0f-4290-b591-a3f239470ba9","Type":"ContainerStarted","Data":"7ed1d3f3ae7401f5f895e91d1317554e3cc92ca7bde3c2045c743f250cfd49fd"}
Jul 24 18:18:51.510248 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:51.510253    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt" event={"ID":"3a04b486-2a0f-4290-b591-a3f239470ba9","Type":"ContainerStarted","Data":"9855090a473599dcd2faa8d0d70634d3a6127749d4a0bc4253a55821e5b6fe77"}
Jul 24 18:18:51.510472 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:51.510280    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"
Jul 24 18:18:51.512387 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:51.512352    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx" event={"ID":"225bb0ec-6f53-4a7c-9412-2c32a3fc1e37","Type":"ContainerStarted","Data":"a2372010198ac83d35065a972111038d476758d91c37f5b432ab70a08378a3cb"}
Jul 24 18:18:51.512506 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:51.512423    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx"
Jul 24 18:18:51.512506 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:51.512445    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx" event={"ID":"225bb0ec-6f53-4a7c-9412-2c32a3fc1e37","Type":"ContainerStarted","Data":"9f1f2ef8d20254f86f43cc58328c9f327120fa0368ab4bc30a68c5d7a4fb20b9"}
Jul 24 18:18:51.534072 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:51.533981    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt" podStartSLOduration=2.12794906 podStartE2EDuration="2.533965585s" totalImagesPullingTime="406.016525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:18:51.5322909 +0000 UTC m=+2154.055649379" watchObservedRunningTime="2026-07-24 18:18:51.533965585 +0000 UTC m=+2154.057324064"
Jul 24 18:18:51.551165 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:51.551114    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx" podStartSLOduration=1.551096055 podStartE2EDuration="1.551096055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:18:51.550271828 +0000 UTC m=+2154.073630310" watchObservedRunningTime="2026-07-24 18:18:51.551096055 +0000 UTC m=+2154.074454535"
Jul 24 18:18:54.011160 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:54.011120    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:54.471262 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:54.471233    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm"
Jul 24 18:18:54.597535 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:54.597499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:57.523251 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:18:57.523222    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"
Jul 24 18:19:01.198231 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:01.198197    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:02.520054 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:02.520021    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx"
Jul 24 18:19:18.004494 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.004456    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"]
Jul 24 18:19:18.005060 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.004713    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd" podUID="1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257" containerName="maas-api" containerID="cri-o://7e5584ee7d0dae0b250d481da9df0b7dfb39cf103fb9272d0ab841e0c2570131" gracePeriod=30
Jul 24 18:19:18.036615 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.036582    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:18.074295 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.074260    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm"]
Jul 24 18:19:18.074547 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.074518    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm" podUID="16bef317-92a6-426b-962a-723c3699d30d" containerName="payload-pre-processing" containerID="cri-o://af1f6928e9bf4fc439cb42ddf38ede16205debf9704a1dfbae0329a1d03505b3" gracePeriod=30
Jul 24 18:19:18.137947 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.137911    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:18.242322 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.242296    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:18.324876 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.324852    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"
Jul 24 18:19:18.360740 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.360719    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm"
Jul 24 18:19:18.426396 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.426361    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-maas-api-tls\") pod \"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257\" (UID: \"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257\") "
Jul 24 18:19:18.426586 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.426439    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th8hc\" (UniqueName: \"kubernetes.io/projected/16bef317-92a6-426b-962a-723c3699d30d-kube-api-access-th8hc\") pod \"16bef317-92a6-426b-962a-723c3699d30d\" (UID: \"16bef317-92a6-426b-962a-723c3699d30d\") "
Jul 24 18:19:18.426586 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.426565    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcbzq\" (UniqueName: \"kubernetes.io/projected/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-kube-api-access-rcbzq\") pod \"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257\" (UID: \"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257\") "
Jul 24 18:19:18.426707 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.426621    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16bef317-92a6-426b-962a-723c3699d30d-plugins-config-volume\") pod \"16bef317-92a6-426b-962a-723c3699d30d\" (UID: \"16bef317-92a6-426b-962a-723c3699d30d\") "
Jul 24 18:19:18.426707 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.426641    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-openshift-service-ca\") pod \"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257\" (UID: \"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257\") "
Jul 24 18:19:18.427064 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.427036    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16bef317-92a6-426b-962a-723c3699d30d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "16bef317-92a6-426b-962a-723c3699d30d" (UID: "16bef317-92a6-426b-962a-723c3699d30d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:19:18.427175 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.427065    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257" (UID: "1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:19:18.428639 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.428616    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-kube-api-access-rcbzq" (OuterVolumeSpecName: "kube-api-access-rcbzq") pod "1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257" (UID: "1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257"). InnerVolumeSpecName "kube-api-access-rcbzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:19:18.428716 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.428647    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257" (UID: "1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:19:18.428716 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.428655    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16bef317-92a6-426b-962a-723c3699d30d-kube-api-access-th8hc" (OuterVolumeSpecName: "kube-api-access-th8hc") pod "16bef317-92a6-426b-962a-723c3699d30d" (UID: "16bef317-92a6-426b-962a-723c3699d30d"). InnerVolumeSpecName "kube-api-access-th8hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:19:18.528153 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.528120    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-openshift-service-ca\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:19:18.528153 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.528146    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-maas-api-tls\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:19:18.528153 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.528155    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th8hc\" (UniqueName: \"kubernetes.io/projected/16bef317-92a6-426b-962a-723c3699d30d-kube-api-access-th8hc\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:19:18.528153 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.528164    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcbzq\" (UniqueName: \"kubernetes.io/projected/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257-kube-api-access-rcbzq\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:19:18.528428 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.528172    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16bef317-92a6-426b-962a-723c3699d30d-plugins-config-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:19:18.661116 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.661075    2578 generic.go:356] "Generic (PLEG): container finished" podID="1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257" containerID="7e5584ee7d0dae0b250d481da9df0b7dfb39cf103fb9272d0ab841e0c2570131" exitCode=0
Jul 24 18:19:18.661329 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.661127    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"
Jul 24 18:19:18.661329 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.661166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd" event={"ID":"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257","Type":"ContainerDied","Data":"7e5584ee7d0dae0b250d481da9df0b7dfb39cf103fb9272d0ab841e0c2570131"}
Jul 24 18:19:18.661329 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.661232    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd" event={"ID":"1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257","Type":"ContainerDied","Data":"5f35254ae76fb152ceaaef260505b8fce69a36da31c391366cd128d1f8a62f67"}
Jul 24 18:19:18.661329 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.661248    2578 scope.go:117] "RemoveContainer" containerID="7e5584ee7d0dae0b250d481da9df0b7dfb39cf103fb9272d0ab841e0c2570131"
Jul 24 18:19:18.662553 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.662528    2578 generic.go:356] "Generic (PLEG): container finished" podID="16bef317-92a6-426b-962a-723c3699d30d" containerID="af1f6928e9bf4fc439cb42ddf38ede16205debf9704a1dfbae0329a1d03505b3" exitCode=0
Jul 24 18:19:18.662687 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.662560    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm" event={"ID":"16bef317-92a6-426b-962a-723c3699d30d","Type":"ContainerDied","Data":"af1f6928e9bf4fc439cb42ddf38ede16205debf9704a1dfbae0329a1d03505b3"}
Jul 24 18:19:18.662687 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.662575    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm"
Jul 24 18:19:18.662687 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.662589    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm" event={"ID":"16bef317-92a6-426b-962a-723c3699d30d","Type":"ContainerDied","Data":"a43c1d5a5f4dcb2cf9eb21925949aeadbdedc18f7720bd1041b0f61aaf52301e"}
Jul 24 18:19:18.671599 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.671580    2578 scope.go:117] "RemoveContainer" containerID="7e5584ee7d0dae0b250d481da9df0b7dfb39cf103fb9272d0ab841e0c2570131"
Jul 24 18:19:18.671902 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:19:18.671883    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e5584ee7d0dae0b250d481da9df0b7dfb39cf103fb9272d0ab841e0c2570131\": container with ID starting with 7e5584ee7d0dae0b250d481da9df0b7dfb39cf103fb9272d0ab841e0c2570131 not found: ID does not exist" containerID="7e5584ee7d0dae0b250d481da9df0b7dfb39cf103fb9272d0ab841e0c2570131"
Jul 24 18:19:18.671977 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.671910    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e5584ee7d0dae0b250d481da9df0b7dfb39cf103fb9272d0ab841e0c2570131"} err="failed to get container status \"7e5584ee7d0dae0b250d481da9df0b7dfb39cf103fb9272d0ab841e0c2570131\": rpc error: code = NotFound desc = could not find container \"7e5584ee7d0dae0b250d481da9df0b7dfb39cf103fb9272d0ab841e0c2570131\": container with ID starting with 7e5584ee7d0dae0b250d481da9df0b7dfb39cf103fb9272d0ab841e0c2570131 not found: ID does not exist"
Jul 24 18:19:18.671977 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.671928    2578 scope.go:117] "RemoveContainer" containerID="af1f6928e9bf4fc439cb42ddf38ede16205debf9704a1dfbae0329a1d03505b3"
Jul 24 18:19:18.682159 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.682099    2578 scope.go:117] "RemoveContainer" containerID="af1f6928e9bf4fc439cb42ddf38ede16205debf9704a1dfbae0329a1d03505b3"
Jul 24 18:19:18.682581 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:19:18.682526    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1f6928e9bf4fc439cb42ddf38ede16205debf9704a1dfbae0329a1d03505b3\": container with ID starting with af1f6928e9bf4fc439cb42ddf38ede16205debf9704a1dfbae0329a1d03505b3 not found: ID does not exist" containerID="af1f6928e9bf4fc439cb42ddf38ede16205debf9704a1dfbae0329a1d03505b3"
Jul 24 18:19:18.682686 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.682563    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1f6928e9bf4fc439cb42ddf38ede16205debf9704a1dfbae0329a1d03505b3"} err="failed to get container status \"af1f6928e9bf4fc439cb42ddf38ede16205debf9704a1dfbae0329a1d03505b3\": rpc error: code = NotFound desc = could not find container \"af1f6928e9bf4fc439cb42ddf38ede16205debf9704a1dfbae0329a1d03505b3\": container with ID starting with af1f6928e9bf4fc439cb42ddf38ede16205debf9704a1dfbae0329a1d03505b3 not found: ID does not exist"
Jul 24 18:19:18.685146 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.685122    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"]
Jul 24 18:19:18.689214 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.689192    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab4b1a-cb97944ff-sqmzd"]
Jul 24 18:19:18.709189 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.702930    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm"]
Jul 24 18:19:18.712760 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:18.712735    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ab4b1a-59d97cc8c6-vh2nm"]
Jul 24 18:19:19.967248 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:19.967216    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16bef317-92a6-426b-962a-723c3699d30d" path="/var/lib/kubelet/pods/16bef317-92a6-426b-962a-723c3699d30d/volumes"
Jul 24 18:19:19.967630 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:19.967616    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257" path="/var/lib/kubelet/pods/1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257/volumes"
Jul 24 18:19:22.275689 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:22.275657    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:23.597448 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:23.597414    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:28.491463 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:28.491427    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:28.589838 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:28.589807    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:28.767776 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:28.767696    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:29.014970 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:29.014938    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:36.066151 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:36.066119    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:36.133044 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:36.133003    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:44.613373 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:44.613331    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"]
Jul 24 18:19:44.614282 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:44.614185    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt" podUID="3a04b486-2a0f-4290-b591-a3f239470ba9" containerName="maas-api" containerID="cri-o://7ed1d3f3ae7401f5f895e91d1317554e3cc92ca7bde3c2045c743f250cfd49fd" gracePeriod=30
Jul 24 18:19:44.634708 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:44.634682    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:44.672056 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:44.672021    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx"]
Jul 24 18:19:44.672621 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:44.672581    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx" podUID="225bb0ec-6f53-4a7c-9412-2c32a3fc1e37" containerName="payload-pre-processing" containerID="cri-o://a2372010198ac83d35065a972111038d476758d91c37f5b432ab70a08378a3cb" gracePeriod=30
Jul 24 18:19:44.734411 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:44.734382    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:44.804479 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:44.804447    2578 generic.go:356] "Generic (PLEG): container finished" podID="225bb0ec-6f53-4a7c-9412-2c32a3fc1e37" containerID="a2372010198ac83d35065a972111038d476758d91c37f5b432ab70a08378a3cb" exitCode=0
Jul 24 18:19:44.804910 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:44.804494    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx" event={"ID":"225bb0ec-6f53-4a7c-9412-2c32a3fc1e37","Type":"ContainerDied","Data":"a2372010198ac83d35065a972111038d476758d91c37f5b432ab70a08378a3cb"}
Jul 24 18:19:44.806592 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:44.806571    2578 generic.go:356] "Generic (PLEG): container finished" podID="3a04b486-2a0f-4290-b591-a3f239470ba9" containerID="7ed1d3f3ae7401f5f895e91d1317554e3cc92ca7bde3c2045c743f250cfd49fd" exitCode=0
Jul 24 18:19:44.806719 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:44.806619    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt" event={"ID":"3a04b486-2a0f-4290-b591-a3f239470ba9","Type":"ContainerDied","Data":"7ed1d3f3ae7401f5f895e91d1317554e3cc92ca7bde3c2045c743f250cfd49fd"}
Jul 24 18:19:44.835216 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:44.834877    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:44.956559 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:44.956530    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"
Jul 24 18:19:44.976514 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:44.976490    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx"
Jul 24 18:19:45.076029 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.075993    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225bb0ec-6f53-4a7c-9412-2c32a3fc1e37-plugins-config-volume\") pod \"225bb0ec-6f53-4a7c-9412-2c32a3fc1e37\" (UID: \"225bb0ec-6f53-4a7c-9412-2c32a3fc1e37\") "
Jul 24 18:19:45.076029 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.076039    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a04b486-2a0f-4290-b591-a3f239470ba9-maas-api-tls\") pod \"3a04b486-2a0f-4290-b591-a3f239470ba9\" (UID: \"3a04b486-2a0f-4290-b591-a3f239470ba9\") "
Jul 24 18:19:45.076263 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.076068    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bh87\" (UniqueName: \"kubernetes.io/projected/3a04b486-2a0f-4290-b591-a3f239470ba9-kube-api-access-2bh87\") pod \"3a04b486-2a0f-4290-b591-a3f239470ba9\" (UID: \"3a04b486-2a0f-4290-b591-a3f239470ba9\") "
Jul 24 18:19:45.076263 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.076101    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jxbn\" (UniqueName: \"kubernetes.io/projected/225bb0ec-6f53-4a7c-9412-2c32a3fc1e37-kube-api-access-2jxbn\") pod \"225bb0ec-6f53-4a7c-9412-2c32a3fc1e37\" (UID: \"225bb0ec-6f53-4a7c-9412-2c32a3fc1e37\") "
Jul 24 18:19:45.076263 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.076254    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a04b486-2a0f-4290-b591-a3f239470ba9-openshift-service-ca\") pod \"3a04b486-2a0f-4290-b591-a3f239470ba9\" (UID: \"3a04b486-2a0f-4290-b591-a3f239470ba9\") "
Jul 24 18:19:45.076484 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.076453    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/225bb0ec-6f53-4a7c-9412-2c32a3fc1e37-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "225bb0ec-6f53-4a7c-9412-2c32a3fc1e37" (UID: "225bb0ec-6f53-4a7c-9412-2c32a3fc1e37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:19:45.076672 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.076648    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a04b486-2a0f-4290-b591-a3f239470ba9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3a04b486-2a0f-4290-b591-a3f239470ba9" (UID: "3a04b486-2a0f-4290-b591-a3f239470ba9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:19:45.078276 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.078253    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a04b486-2a0f-4290-b591-a3f239470ba9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3a04b486-2a0f-4290-b591-a3f239470ba9" (UID: "3a04b486-2a0f-4290-b591-a3f239470ba9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:19:45.078365 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.078288    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a04b486-2a0f-4290-b591-a3f239470ba9-kube-api-access-2bh87" (OuterVolumeSpecName: "kube-api-access-2bh87") pod "3a04b486-2a0f-4290-b591-a3f239470ba9" (UID: "3a04b486-2a0f-4290-b591-a3f239470ba9"). InnerVolumeSpecName "kube-api-access-2bh87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:19:45.078365 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.078298    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225bb0ec-6f53-4a7c-9412-2c32a3fc1e37-kube-api-access-2jxbn" (OuterVolumeSpecName: "kube-api-access-2jxbn") pod "225bb0ec-6f53-4a7c-9412-2c32a3fc1e37" (UID: "225bb0ec-6f53-4a7c-9412-2c32a3fc1e37"). InnerVolumeSpecName "kube-api-access-2jxbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:19:45.177949 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.177858    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a04b486-2a0f-4290-b591-a3f239470ba9-openshift-service-ca\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:19:45.177949 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.177892    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225bb0ec-6f53-4a7c-9412-2c32a3fc1e37-plugins-config-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:19:45.177949 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.177903    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a04b486-2a0f-4290-b591-a3f239470ba9-maas-api-tls\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:19:45.177949 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.177913    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bh87\" (UniqueName: \"kubernetes.io/projected/3a04b486-2a0f-4290-b591-a3f239470ba9-kube-api-access-2bh87\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:19:45.177949 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.177921    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jxbn\" (UniqueName: \"kubernetes.io/projected/225bb0ec-6f53-4a7c-9412-2c32a3fc1e37-kube-api-access-2jxbn\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:19:45.813061 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.813031    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"
Jul 24 18:19:45.813557 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.813061    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt" event={"ID":"3a04b486-2a0f-4290-b591-a3f239470ba9","Type":"ContainerDied","Data":"9855090a473599dcd2faa8d0d70634d3a6127749d4a0bc4253a55821e5b6fe77"}
Jul 24 18:19:45.813557 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.813138    2578 scope.go:117] "RemoveContainer" containerID="7ed1d3f3ae7401f5f895e91d1317554e3cc92ca7bde3c2045c743f250cfd49fd"
Jul 24 18:19:45.814640 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.814620    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx"
Jul 24 18:19:45.814768 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.814619    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx" event={"ID":"225bb0ec-6f53-4a7c-9412-2c32a3fc1e37","Type":"ContainerDied","Data":"9f1f2ef8d20254f86f43cc58328c9f327120fa0368ab4bc30a68c5d7a4fb20b9"}
Jul 24 18:19:45.822377 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.822362    2578 scope.go:117] "RemoveContainer" containerID="a2372010198ac83d35065a972111038d476758d91c37f5b432ab70a08378a3cb"
Jul 24 18:19:45.837857 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.837831    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"]
Jul 24 18:19:45.842088 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.842066    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ac2de9-688b744b59-f8nnt"]
Jul 24 18:19:45.853717 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.853693    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx"]
Jul 24 18:19:45.858403 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.858381    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ac2de9-85fb58d788-47pbx"]
Jul 24 18:19:45.968171 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.968093    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="225bb0ec-6f53-4a7c-9412-2c32a3fc1e37" path="/var/lib/kubelet/pods/225bb0ec-6f53-4a7c-9412-2c32a3fc1e37/volumes"
Jul 24 18:19:45.968492 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:45.968478    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a04b486-2a0f-4290-b591-a3f239470ba9" path="/var/lib/kubelet/pods/3a04b486-2a0f-4290-b591-a3f239470ba9/volumes"
Jul 24 18:19:49.254517 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:49.254480    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:49.665894 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:49.665864    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:55.487347 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:55.487311    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:55.589181 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:55.589149    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:55.767137 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:55.767054    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:56.007233 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:56.007207    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:56.381001 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:56.380960    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:56.823175 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:56.823145    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:56.983448 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:19:56.983415    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:00.065019 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.064985    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:00.147221 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.147183    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"]
Jul 24 18:20:00.147722 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.147701    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16bef317-92a6-426b-962a-723c3699d30d" containerName="payload-pre-processing"
Jul 24 18:20:00.147811 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.147725    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="16bef317-92a6-426b-962a-723c3699d30d" containerName="payload-pre-processing"
Jul 24 18:20:00.147811 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.147756    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a04b486-2a0f-4290-b591-a3f239470ba9" containerName="maas-api"
Jul 24 18:20:00.147811 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.147765    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a04b486-2a0f-4290-b591-a3f239470ba9" containerName="maas-api"
Jul 24 18:20:00.147811 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.147806    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="225bb0ec-6f53-4a7c-9412-2c32a3fc1e37" containerName="payload-pre-processing"
Jul 24 18:20:00.147944 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.147816    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="225bb0ec-6f53-4a7c-9412-2c32a3fc1e37" containerName="payload-pre-processing"
Jul 24 18:20:00.147944 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.147835    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257" containerName="maas-api"
Jul 24 18:20:00.147944 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.147844    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257" containerName="maas-api"
Jul 24 18:20:00.147944 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.147934    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f7f0f0e-8cb1-4fbc-861d-aa4e6e6e8257" containerName="maas-api"
Jul 24 18:20:00.148069 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.147948    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="16bef317-92a6-426b-962a-723c3699d30d" containerName="payload-pre-processing"
Jul 24 18:20:00.148069 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.147956    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="225bb0ec-6f53-4a7c-9412-2c32a3fc1e37" containerName="payload-pre-processing"
Jul 24 18:20:00.148069 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.147963    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a04b486-2a0f-4290-b591-a3f239470ba9" containerName="maas-api"
Jul 24 18:20:00.151362 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.151345    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"
Jul 24 18:20:00.154682 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.154661    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-2542df-serving-cert\""
Jul 24 18:20:00.164703 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.164679    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"]
Jul 24 18:20:00.172725 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.172701    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:00.211694 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.211653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca089800-66f3-482f-92db-917bc1868fb0-maas-api-tls\") pod \"maas-api-e2e-shared-a-2542df-6764947cb-l6h7l\" (UID: \"ca089800-66f3-482f-92db-917bc1868fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"
Jul 24 18:20:00.211933 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.211862    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqqm7\" (UniqueName: \"kubernetes.io/projected/ca089800-66f3-482f-92db-917bc1868fb0-kube-api-access-tqqm7\") pod \"maas-api-e2e-shared-a-2542df-6764947cb-l6h7l\" (UID: \"ca089800-66f3-482f-92db-917bc1868fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"
Jul 24 18:20:00.212025 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.211944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca089800-66f3-482f-92db-917bc1868fb0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2542df-6764947cb-l6h7l\" (UID: \"ca089800-66f3-482f-92db-917bc1868fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"
Jul 24 18:20:00.266624 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.266593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:00.312488 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.312450    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca089800-66f3-482f-92db-917bc1868fb0-maas-api-tls\") pod \"maas-api-e2e-shared-a-2542df-6764947cb-l6h7l\" (UID: \"ca089800-66f3-482f-92db-917bc1868fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"
Jul 24 18:20:00.312671 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.312507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqqm7\" (UniqueName: \"kubernetes.io/projected/ca089800-66f3-482f-92db-917bc1868fb0-kube-api-access-tqqm7\") pod \"maas-api-e2e-shared-a-2542df-6764947cb-l6h7l\" (UID: \"ca089800-66f3-482f-92db-917bc1868fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"
Jul 24 18:20:00.312671 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:20:00.312599    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-serving-cert: secret "maas-api-e2e-shared-a-2542df-serving-cert" not found
Jul 24 18:20:00.312671 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:20:00.312666    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca089800-66f3-482f-92db-917bc1868fb0-maas-api-tls podName:ca089800-66f3-482f-92db-917bc1868fb0 nodeName:}" failed. No retries permitted until 2026-07-24 18:20:00.812646526 +0000 UTC m=+2223.336004998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ca089800-66f3-482f-92db-917bc1868fb0-maas-api-tls") pod "maas-api-e2e-shared-a-2542df-6764947cb-l6h7l" (UID: "ca089800-66f3-482f-92db-917bc1868fb0") : secret "maas-api-e2e-shared-a-2542df-serving-cert" not found
Jul 24 18:20:00.312838 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.312681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca089800-66f3-482f-92db-917bc1868fb0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2542df-6764947cb-l6h7l\" (UID: \"ca089800-66f3-482f-92db-917bc1868fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"
Jul 24 18:20:00.313423 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.313401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca089800-66f3-482f-92db-917bc1868fb0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2542df-6764947cb-l6h7l\" (UID: \"ca089800-66f3-482f-92db-917bc1868fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"
Jul 24 18:20:00.324612 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.324535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqqm7\" (UniqueName: \"kubernetes.io/projected/ca089800-66f3-482f-92db-917bc1868fb0-kube-api-access-tqqm7\") pod \"maas-api-e2e-shared-a-2542df-6764947cb-l6h7l\" (UID: \"ca089800-66f3-482f-92db-917bc1868fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"
Jul 24 18:20:00.506024 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.505986    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:00.750995 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.750911    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:00.819393 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.819353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca089800-66f3-482f-92db-917bc1868fb0-maas-api-tls\") pod \"maas-api-e2e-shared-a-2542df-6764947cb-l6h7l\" (UID: \"ca089800-66f3-482f-92db-917bc1868fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"
Jul 24 18:20:00.821735 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.821701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca089800-66f3-482f-92db-917bc1868fb0-maas-api-tls\") pod \"maas-api-e2e-shared-a-2542df-6764947cb-l6h7l\" (UID: \"ca089800-66f3-482f-92db-917bc1868fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"
Jul 24 18:20:00.850567 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.850530    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:00.897687 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.897655    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84"]
Jul 24 18:20:00.900004 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.899984    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84"
Jul 24 18:20:00.907359 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.907333    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-2542df-dockercfg-4x8rz\""
Jul 24 18:20:00.907489 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.907333    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-2542df\""
Jul 24 18:20:00.914759 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.914735    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84"]
Jul 24 18:20:00.920063 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.920041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zj9b\" (UniqueName: \"kubernetes.io/projected/fa0c01c6-ed03-498b-a386-44c85d71e601-kube-api-access-2zj9b\") pod \"payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84\" (UID: \"fa0c01c6-ed03-498b-a386-44c85d71e601\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84"
Jul 24 18:20:00.920176 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.920075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0c01c6-ed03-498b-a386-44c85d71e601-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84\" (UID: \"fa0c01c6-ed03-498b-a386-44c85d71e601\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84"
Jul 24 18:20:00.937189 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.937158    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm"]
Jul 24 18:20:00.940291 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.940273    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm"
Jul 24 18:20:00.951848 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:00.951823    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm"]
Jul 24 18:20:01.020855 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.020741    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846247f7-38c6-438b-b04b-8681967a7124-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm\" (UID: \"846247f7-38c6-438b-b04b-8681967a7124\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm"
Jul 24 18:20:01.021028 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.020868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zj9b\" (UniqueName: \"kubernetes.io/projected/fa0c01c6-ed03-498b-a386-44c85d71e601-kube-api-access-2zj9b\") pod \"payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84\" (UID: \"fa0c01c6-ed03-498b-a386-44c85d71e601\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84"
Jul 24 18:20:01.021028 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.020914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0c01c6-ed03-498b-a386-44c85d71e601-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84\" (UID: \"fa0c01c6-ed03-498b-a386-44c85d71e601\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84"
Jul 24 18:20:01.021028 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.020977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b7wv\" (UniqueName: \"kubernetes.io/projected/846247f7-38c6-438b-b04b-8681967a7124-kube-api-access-5b7wv\") pod \"payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm\" (UID: \"846247f7-38c6-438b-b04b-8681967a7124\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm"
Jul 24 18:20:01.021650 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.021626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0c01c6-ed03-498b-a386-44c85d71e601-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84\" (UID: \"fa0c01c6-ed03-498b-a386-44c85d71e601\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84"
Jul 24 18:20:01.033275 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.033243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zj9b\" (UniqueName: \"kubernetes.io/projected/fa0c01c6-ed03-498b-a386-44c85d71e601-kube-api-access-2zj9b\") pod \"payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84\" (UID: \"fa0c01c6-ed03-498b-a386-44c85d71e601\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84"
Jul 24 18:20:01.061187 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.061148    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"
Jul 24 18:20:01.121989 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.121952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b7wv\" (UniqueName: \"kubernetes.io/projected/846247f7-38c6-438b-b04b-8681967a7124-kube-api-access-5b7wv\") pod \"payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm\" (UID: \"846247f7-38c6-438b-b04b-8681967a7124\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm"
Jul 24 18:20:01.122302 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.122076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846247f7-38c6-438b-b04b-8681967a7124-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm\" (UID: \"846247f7-38c6-438b-b04b-8681967a7124\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm"
Jul 24 18:20:01.123100 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.123074    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846247f7-38c6-438b-b04b-8681967a7124-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm\" (UID: \"846247f7-38c6-438b-b04b-8681967a7124\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm"
Jul 24 18:20:01.133030 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.132960    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b7wv\" (UniqueName: \"kubernetes.io/projected/846247f7-38c6-438b-b04b-8681967a7124-kube-api-access-5b7wv\") pod \"payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm\" (UID: \"846247f7-38c6-438b-b04b-8681967a7124\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm"
Jul 24 18:20:01.211573 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.211530    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84"
Jul 24 18:20:01.250839 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.250803    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm"
Jul 24 18:20:01.378009 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.377977    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84"]
Jul 24 18:20:01.438291 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.438262    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"]
Jul 24 18:20:01.441552 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.441524    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm"]
Jul 24 18:20:01.443867 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:20:01.443380    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca089800_66f3_482f_92db_917bc1868fb0.slice/crio-6db1f673200f64b09d02883dc8d8729d31d0af205ff9e02ad21eabc2898d05b2 WatchSource:0}: Error finding container 6db1f673200f64b09d02883dc8d8729d31d0af205ff9e02ad21eabc2898d05b2: Status 404 returned error can't find the container with id 6db1f673200f64b09d02883dc8d8729d31d0af205ff9e02ad21eabc2898d05b2
Jul 24 18:20:01.444010 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:20:01.443911    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846247f7_38c6_438b_b04b_8681967a7124.slice/crio-63c5f02ee7febe6306cda627a91e3daaa363239b5170ad51f604aadd6309aa87 WatchSource:0}: Error finding container 63c5f02ee7febe6306cda627a91e3daaa363239b5170ad51f604aadd6309aa87: Status 404 returned error can't find the container with id 63c5f02ee7febe6306cda627a91e3daaa363239b5170ad51f604aadd6309aa87
Jul 24 18:20:01.897421 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.897380    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l" event={"ID":"ca089800-66f3-482f-92db-917bc1868fb0","Type":"ContainerStarted","Data":"6db1f673200f64b09d02883dc8d8729d31d0af205ff9e02ad21eabc2898d05b2"}
Jul 24 18:20:01.899245 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.899211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84" event={"ID":"fa0c01c6-ed03-498b-a386-44c85d71e601","Type":"ContainerStarted","Data":"0c4a31768040ba61b345f074ac7402971d701519132e809d5061411c0ffdff03"}
Jul 24 18:20:01.899397 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.899273    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84" event={"ID":"fa0c01c6-ed03-498b-a386-44c85d71e601","Type":"ContainerStarted","Data":"720b4eab83dac4a1b5099c5830ae70b673d22000c4438c70e579952260ba57b8"}
Jul 24 18:20:01.899451 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.899397    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84"
Jul 24 18:20:01.900869 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.900838    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm" event={"ID":"846247f7-38c6-438b-b04b-8681967a7124","Type":"ContainerStarted","Data":"759ff495741ab57757ced944fc7df40d8f6fe164c95648be72cf241f44861754"}
Jul 24 18:20:01.900990 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.900879    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm" event={"ID":"846247f7-38c6-438b-b04b-8681967a7124","Type":"ContainerStarted","Data":"63c5f02ee7febe6306cda627a91e3daaa363239b5170ad51f604aadd6309aa87"}
Jul 24 18:20:01.900990 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.900919    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm"
Jul 24 18:20:01.922609 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.922544    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84" podStartSLOduration=1.922520704 podStartE2EDuration="1.922520704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:20:01.920201847 +0000 UTC m=+2224.443560327" watchObservedRunningTime="2026-07-24 18:20:01.922520704 +0000 UTC m=+2224.445879253"
Jul 24 18:20:01.940751 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:01.940703    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm" podStartSLOduration=1.940688036 podStartE2EDuration="1.940688036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:20:01.939081178 +0000 UTC m=+2224.462439657" watchObservedRunningTime="2026-07-24 18:20:01.940688036 +0000 UTC m=+2224.464046554"
Jul 24 18:20:02.908415 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:02.908369    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l" event={"ID":"ca089800-66f3-482f-92db-917bc1868fb0","Type":"ContainerStarted","Data":"8b14872417568ebe76b21c64f63891ea357d552a250081bd2ddbcef4c43ae87e"}
Jul 24 18:20:02.908947 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:02.908660    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"
Jul 24 18:20:02.930528 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:02.930473    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l" podStartSLOduration=2.424247637 podStartE2EDuration="2.930455492s" totalImagesPullingTime="506.207855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:20:02.9281411 +0000 UTC m=+2225.451499580" watchObservedRunningTime="2026-07-24 18:20:02.930455492 +0000 UTC m=+2225.453813970"
Jul 24 18:20:04.631714 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:04.631670    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:04.963302 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:04.963212    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:05.419914 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:05.419859    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:05.621703 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:05.621666    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:08.119612 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:08.119577    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:08.207168 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:08.207131    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"]
Jul 24 18:20:08.209692 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:08.209672    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"
Jul 24 18:20:08.213241 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:08.213210    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-f556c1-serving-cert\""
Jul 24 18:20:08.234930 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:08.234895    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"]
Jul 24 18:20:08.256953 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:08.256918    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:08.294960 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:08.294922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-maas-api-tls\") pod \"maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw\" (UID: \"9f32a8b6-14a6-4d7f-b647-0140590a6ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"
Jul 24 18:20:08.295135 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:08.295074    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r5tj\" (UniqueName: \"kubernetes.io/projected/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-kube-api-access-5r5tj\") pod \"maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw\" (UID: \"9f32a8b6-14a6-4d7f-b647-0140590a6ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"
Jul 24 18:20:08.295179 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:08.295130    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw\" (UID: \"9f32a8b6-14a6-4d7f-b647-0140590a6ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"
Jul 24 18:20:08.361909 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:08.361876    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:08.395969 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:08.395885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r5tj\" (UniqueName: \"kubernetes.io/projected/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-kube-api-access-5r5tj\") pod \"maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw\" (UID: \"9f32a8b6-14a6-4d7f-b647-0140590a6ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"
Jul 24 18:20:08.395969 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:08.395936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw\" (UID: \"9f32a8b6-14a6-4d7f-b647-0140590a6ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"
Jul 24 18:20:08.396162 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:08.395987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-maas-api-tls\") pod \"maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw\" (UID: \"9f32a8b6-14a6-4d7f-b647-0140590a6ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"
Jul 24 18:20:08.396162 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:20:08.396093    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-serving-cert: secret "maas-api-e2e-shared-b-f556c1-serving-cert" not found
Jul 24 18:20:08.396162 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:20:08.396155    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-maas-api-tls podName:9f32a8b6-14a6-4d7f-b647-0140590a6ce8 nodeName:}" failed. No retries permitted until 2026-07-24 18:20:08.896136621 +0000 UTC m=+2231.419495077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-maas-api-tls") pod "maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw" (UID: "9f32a8b6-14a6-4d7f-b647-0140590a6ce8") : secret "maas-api-e2e-shared-b-f556c1-serving-cert" not found
Jul 24 18:20:08.396814 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:08.396773    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw\" (UID: \"9f32a8b6-14a6-4d7f-b647-0140590a6ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"
Jul 24 18:20:08.418351 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:08.418321    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r5tj\" (UniqueName: \"kubernetes.io/projected/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-kube-api-access-5r5tj\") pod \"maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw\" (UID: \"9f32a8b6-14a6-4d7f-b647-0140590a6ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"
Jul 24 18:20:08.611841 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:08.611806    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:08.900401 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:08.900353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-maas-api-tls\") pod \"maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw\" (UID: \"9f32a8b6-14a6-4d7f-b647-0140590a6ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"
Jul 24 18:20:08.902965 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:08.902931    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-maas-api-tls\") pod \"maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw\" (UID: \"9f32a8b6-14a6-4d7f-b647-0140590a6ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"
Jul 24 18:20:08.921638 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:08.921611    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"
Jul 24 18:20:09.012216 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.012187    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:09.111208 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.111167    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:09.119987 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.119956    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"
Jul 24 18:20:09.222244 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.220240    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v"]
Jul 24 18:20:09.226650 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.226627    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v"
Jul 24 18:20:09.231397 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.230686    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-f556c1-dockercfg-jl66d\""
Jul 24 18:20:09.231397 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.230985    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-f556c1\""
Jul 24 18:20:09.231670 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.231514    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v"]
Jul 24 18:20:09.283267 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.281481    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"]
Jul 24 18:20:09.285146 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:20:09.285113    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f32a8b6_14a6_4d7f_b647_0140590a6ce8.slice/crio-1b472c02955ed5ceae8b284a0c5c4d84ff5ad646ee75f06c31c54a4c8f7c5d3e WatchSource:0}: Error finding container 1b472c02955ed5ceae8b284a0c5c4d84ff5ad646ee75f06c31c54a4c8f7c5d3e: Status 404 returned error can't find the container with id 1b472c02955ed5ceae8b284a0c5c4d84ff5ad646ee75f06c31c54a4c8f7c5d3e
Jul 24 18:20:09.305974 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.305922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dls8n\" (UniqueName: \"kubernetes.io/projected/b116a61e-f964-4bb7-860c-7f3e6a799af9-kube-api-access-dls8n\") pod \"payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v\" (UID: \"b116a61e-f964-4bb7-860c-7f3e6a799af9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v"
Jul 24 18:20:09.306171 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.306127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b116a61e-f964-4bb7-860c-7f3e6a799af9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v\" (UID: \"b116a61e-f964-4bb7-860c-7f3e6a799af9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v"
Jul 24 18:20:09.406826 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.406766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b116a61e-f964-4bb7-860c-7f3e6a799af9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v\" (UID: \"b116a61e-f964-4bb7-860c-7f3e6a799af9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v"
Jul 24 18:20:09.407006 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.406870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dls8n\" (UniqueName: \"kubernetes.io/projected/b116a61e-f964-4bb7-860c-7f3e6a799af9-kube-api-access-dls8n\") pod \"payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v\" (UID: \"b116a61e-f964-4bb7-860c-7f3e6a799af9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v"
Jul 24 18:20:09.407602 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.407580    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b116a61e-f964-4bb7-860c-7f3e6a799af9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v\" (UID: \"b116a61e-f964-4bb7-860c-7f3e6a799af9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v"
Jul 24 18:20:09.416088 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.416062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dls8n\" (UniqueName: \"kubernetes.io/projected/b116a61e-f964-4bb7-860c-7f3e6a799af9-kube-api-access-dls8n\") pod \"payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v\" (UID: \"b116a61e-f964-4bb7-860c-7f3e6a799af9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v"
Jul 24 18:20:09.542417 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.542313    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v"
Jul 24 18:20:09.693887 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.693856    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v"]
Jul 24 18:20:09.696042 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:20:09.696012    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb116a61e_f964_4bb7_860c_7f3e6a799af9.slice/crio-fe5a72c33d852583e219a6b66450563b55147a90dc7b55b2edab5ab5c83aac75 WatchSource:0}: Error finding container fe5a72c33d852583e219a6b66450563b55147a90dc7b55b2edab5ab5c83aac75: Status 404 returned error can't find the container with id fe5a72c33d852583e219a6b66450563b55147a90dc7b55b2edab5ab5c83aac75
Jul 24 18:20:09.947993 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.947956    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v" event={"ID":"b116a61e-f964-4bb7-860c-7f3e6a799af9","Type":"ContainerStarted","Data":"249e8049654a33235213b1518b97a392d3494637d7cfe35549cd3e26c6447247"}
Jul 24 18:20:09.948197 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.948020    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v"
Jul 24 18:20:09.948197 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.948041    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v" event={"ID":"b116a61e-f964-4bb7-860c-7f3e6a799af9","Type":"ContainerStarted","Data":"fe5a72c33d852583e219a6b66450563b55147a90dc7b55b2edab5ab5c83aac75"}
Jul 24 18:20:09.949522 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.949499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw" event={"ID":"9f32a8b6-14a6-4d7f-b647-0140590a6ce8","Type":"ContainerStarted","Data":"3f2a9c7f94c82b02e824b631364db6b73073ff98688749bf34b7ad2cefffb87e"}
Jul 24 18:20:09.949646 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.949545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw" event={"ID":"9f32a8b6-14a6-4d7f-b647-0140590a6ce8","Type":"ContainerStarted","Data":"1b472c02955ed5ceae8b284a0c5c4d84ff5ad646ee75f06c31c54a4c8f7c5d3e"}
Jul 24 18:20:09.949646 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.949573    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"
Jul 24 18:20:09.972460 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.972409    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v" podStartSLOduration=0.972390346 podStartE2EDuration="972.390346ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:20:09.970885358 +0000 UTC m=+2232.494243860" watchObservedRunningTime="2026-07-24 18:20:09.972390346 +0000 UTC m=+2232.495748815"
Jul 24 18:20:09.997137 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:09.996502    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw" podStartSLOduration=1.637309863 podStartE2EDuration="1.996480678s" totalImagesPullingTime="359.170815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:20:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:20:09.995902999 +0000 UTC m=+2232.519261477" watchObservedRunningTime="2026-07-24 18:20:09.996480678 +0000 UTC m=+2232.519839157"
Jul 24 18:20:12.647077 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:12.647038    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:12.910931 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:12.910839    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm"
Jul 24 18:20:12.911071 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:12.910934    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84"
Jul 24 18:20:15.959771 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:15.959742    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"
Jul 24 18:20:20.957902 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:20.957821    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v"
Jul 24 18:20:21.804259 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:21.804221    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:55.278720 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:20:55.278686    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:21:10.274657 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:21:10.274620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:22:07.270772 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:22:07.270683    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:22:29.766075 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:22:29.766031    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:22:30.674533 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:22:30.674487    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:22:55.593313 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:22:55.593254    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:22:59.085716 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:22:59.085686    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:23:22.468260 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:23:22.468179    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:23:58.226583 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:23:58.226545    2578 scope.go:117] "RemoveContainer" containerID="5c299e8960512996c23729e58d8b7f056c646a6845105fe1b07013724363c205"
Jul 24 18:24:03.742198 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:24:03.742167    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:24:04.641037 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:24:04.641003    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:24:11.094606 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:24:11.094531    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:24:17.547851 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:24:17.547813    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:24:17.645548 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:24:17.645511    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:24:31.293766 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:24:31.293712    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:24:58.246941 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:24:58.246847    2578 scope.go:117] "RemoveContainer" containerID="fa1855168476f22fc4e6a39a192423c20fe3d21f3fb22ba509772399d594f935"
Jul 24 18:25:15.347741 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:15.347708    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:16.140479 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:16.140447    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:36.179771 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:36.179733    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:36.491990 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:36.491916    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:36.917825 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:36.917764    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:39.433210 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:39.433169    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:39.529847 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:39.529805    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"]
Jul 24 18:25:39.532825 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:39.532773    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:39.534196 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:39.534170    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"
Jul 24 18:25:39.537085 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:39.537062    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d91e94-serving-cert\""
Jul 24 18:25:39.548681 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:39.548656    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"]
Jul 24 18:25:39.632525 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:39.632485    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:39.649450 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:39.649419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb6fb89-9bca-4788-9807-f70d4dc32b2b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv\" (UID: \"deb6fb89-9bca-4788-9807-f70d4dc32b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"
Jul 24 18:25:39.649609 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:39.649494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb6fb89-9bca-4788-9807-f70d4dc32b2b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv\" (UID: \"deb6fb89-9bca-4788-9807-f70d4dc32b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"
Jul 24 18:25:39.649609 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:39.649554    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mhhj\" (UniqueName: \"kubernetes.io/projected/deb6fb89-9bca-4788-9807-f70d4dc32b2b-kube-api-access-9mhhj\") pod \"maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv\" (UID: \"deb6fb89-9bca-4788-9807-f70d4dc32b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"
Jul 24 18:25:39.750936 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:39.750852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb6fb89-9bca-4788-9807-f70d4dc32b2b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv\" (UID: \"deb6fb89-9bca-4788-9807-f70d4dc32b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"
Jul 24 18:25:39.750936 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:39.750917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mhhj\" (UniqueName: \"kubernetes.io/projected/deb6fb89-9bca-4788-9807-f70d4dc32b2b-kube-api-access-9mhhj\") pod \"maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv\" (UID: \"deb6fb89-9bca-4788-9807-f70d4dc32b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"
Jul 24 18:25:39.751144 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:39.750965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb6fb89-9bca-4788-9807-f70d4dc32b2b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv\" (UID: \"deb6fb89-9bca-4788-9807-f70d4dc32b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"
Jul 24 18:25:39.751144 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:25:39.751004    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-serving-cert: secret "maas-api-e2e-ipp-a-d91e94-serving-cert" not found
Jul 24 18:25:39.751144 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:25:39.751083    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb6fb89-9bca-4788-9807-f70d4dc32b2b-maas-api-tls podName:deb6fb89-9bca-4788-9807-f70d4dc32b2b nodeName:}" failed. No retries permitted until 2026-07-24 18:25:40.251066532 +0000 UTC m=+2562.774424987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/deb6fb89-9bca-4788-9807-f70d4dc32b2b-maas-api-tls") pod "maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv" (UID: "deb6fb89-9bca-4788-9807-f70d4dc32b2b") : secret "maas-api-e2e-ipp-a-d91e94-serving-cert" not found
Jul 24 18:25:39.751524 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:39.751506    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb6fb89-9bca-4788-9807-f70d4dc32b2b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv\" (UID: \"deb6fb89-9bca-4788-9807-f70d4dc32b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"
Jul 24 18:25:39.759979 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:39.759950    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mhhj\" (UniqueName: \"kubernetes.io/projected/deb6fb89-9bca-4788-9807-f70d4dc32b2b-kube-api-access-9mhhj\") pod \"maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv\" (UID: \"deb6fb89-9bca-4788-9807-f70d4dc32b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"
Jul 24 18:25:39.903339 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:39.903308    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:40.244801 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.244746    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:40.255879 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.255849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb6fb89-9bca-4788-9807-f70d4dc32b2b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv\" (UID: \"deb6fb89-9bca-4788-9807-f70d4dc32b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"
Jul 24 18:25:40.258275 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.258246    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb6fb89-9bca-4788-9807-f70d4dc32b2b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv\" (UID: \"deb6fb89-9bca-4788-9807-f70d4dc32b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"
Jul 24 18:25:40.445155 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.445114    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"
Jul 24 18:25:40.445593 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.445361    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr"]
Jul 24 18:25:40.448716 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.448697    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr"
Jul 24 18:25:40.451879 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.451857    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d91e94\""
Jul 24 18:25:40.456311 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.456174    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d91e94-dockercfg-4hlll\""
Jul 24 18:25:40.469082 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.469048    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr"]
Jul 24 18:25:40.521595 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.521552    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:40.560296 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.560157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66rp\" (UniqueName: \"kubernetes.io/projected/21231bc0-059e-4466-b88b-a7eac3530a56-kube-api-access-f66rp\") pod \"payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr\" (UID: \"21231bc0-059e-4466-b88b-a7eac3530a56\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr"
Jul 24 18:25:40.560296 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.560212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21231bc0-059e-4466-b88b-a7eac3530a56-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr\" (UID: \"21231bc0-059e-4466-b88b-a7eac3530a56\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr"
Jul 24 18:25:40.610791 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.610743    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"]
Jul 24 18:25:40.613443 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:25:40.613410    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb6fb89_9bca_4788_9807_f70d4dc32b2b.slice/crio-3b1f46714a7f2e28a6cd4d190885b7f4d130e49aaca53e02b93ec6776e84d07b WatchSource:0}: Error finding container 3b1f46714a7f2e28a6cd4d190885b7f4d130e49aaca53e02b93ec6776e84d07b: Status 404 returned error can't find the container with id 3b1f46714a7f2e28a6cd4d190885b7f4d130e49aaca53e02b93ec6776e84d07b
Jul 24 18:25:40.614907 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.614883    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 18:25:40.661295 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.661261    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66rp\" (UniqueName: \"kubernetes.io/projected/21231bc0-059e-4466-b88b-a7eac3530a56-kube-api-access-f66rp\") pod \"payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr\" (UID: \"21231bc0-059e-4466-b88b-a7eac3530a56\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr"
Jul 24 18:25:40.661295 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.661301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21231bc0-059e-4466-b88b-a7eac3530a56-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr\" (UID: \"21231bc0-059e-4466-b88b-a7eac3530a56\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr"
Jul 24 18:25:40.662116 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.662095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21231bc0-059e-4466-b88b-a7eac3530a56-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr\" (UID: \"21231bc0-059e-4466-b88b-a7eac3530a56\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr"
Jul 24 18:25:40.670553 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.670528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66rp\" (UniqueName: \"kubernetes.io/projected/21231bc0-059e-4466-b88b-a7eac3530a56-kube-api-access-f66rp\") pod \"payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr\" (UID: \"21231bc0-059e-4466-b88b-a7eac3530a56\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr"
Jul 24 18:25:40.680869 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.680837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv" event={"ID":"deb6fb89-9bca-4788-9807-f70d4dc32b2b","Type":"ContainerStarted","Data":"3b1f46714a7f2e28a6cd4d190885b7f4d130e49aaca53e02b93ec6776e84d07b"}
Jul 24 18:25:40.780640 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:40.780548    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr"
Jul 24 18:25:41.129877 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:41.129850    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr"]
Jul 24 18:25:41.132403 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:25:41.132370    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21231bc0_059e_4466_b88b_a7eac3530a56.slice/crio-e75d517bec708c954aae299cc850f535826a605be7622cb8010e17c6093905e8 WatchSource:0}: Error finding container e75d517bec708c954aae299cc850f535826a605be7622cb8010e17c6093905e8: Status 404 returned error can't find the container with id e75d517bec708c954aae299cc850f535826a605be7622cb8010e17c6093905e8
Jul 24 18:25:41.688105 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:41.688070    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr" event={"ID":"21231bc0-059e-4466-b88b-a7eac3530a56","Type":"ContainerStarted","Data":"7ab8d908f6049f872e6be0356296b4e35462d0d218abb40b78171732d12688bb"}
Jul 24 18:25:41.688604 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:41.688132    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr" event={"ID":"21231bc0-059e-4466-b88b-a7eac3530a56","Type":"ContainerStarted","Data":"e75d517bec708c954aae299cc850f535826a605be7622cb8010e17c6093905e8"}
Jul 24 18:25:41.688604 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:41.688164    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr"
Jul 24 18:25:41.689620 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:41.689591    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv" event={"ID":"deb6fb89-9bca-4788-9807-f70d4dc32b2b","Type":"ContainerStarted","Data":"73bef9d5cb82de8675ffc75131ad1cfe59d87309235b0a42ec6369b03926e61d"}
Jul 24 18:25:41.689981 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:41.689642    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"
Jul 24 18:25:41.706211 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:41.706160    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr" podStartSLOduration=1.706144556 podStartE2EDuration="1.706144556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:25:41.705233775 +0000 UTC m=+2564.228592253" watchObservedRunningTime="2026-07-24 18:25:41.706144556 +0000 UTC m=+2564.229503079"
Jul 24 18:25:41.726962 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:41.726904    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv" podStartSLOduration=2.201867356 podStartE2EDuration="2.726888388s" totalImagesPullingTime="525.021032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:25:41.724951957 +0000 UTC m=+2564.248310436" watchObservedRunningTime="2026-07-24 18:25:41.726888388 +0000 UTC m=+2564.250246865"
Jul 24 18:25:41.986861 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:41.986754    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:43.953517 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:43.953485    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:47.701144 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:47.701113    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"
Jul 24 18:25:52.697879 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:52.697842    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr"
Jul 24 18:25:54.896717 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:54.896662    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:55.327753 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:55.327717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:55.506736 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:55.506693    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:58.602309 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:58.602267    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:58.666998 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:58.666966    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"]
Jul 24 18:25:58.672507 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:58.672481    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"
Jul 24 18:25:58.675133 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:58.675107    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-651f67-serving-cert\""
Jul 24 18:25:58.684468 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:58.684445    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"]
Jul 24 18:25:58.688032 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:58.687984    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:58.758749 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:58.758716    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:58.843624 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:58.843589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwgz7\" (UniqueName: \"kubernetes.io/projected/dae8ab73-e781-4365-9fe4-c923d7753955-kube-api-access-fwgz7\") pod \"maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf\" (UID: \"dae8ab73-e781-4365-9fe4-c923d7753955\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"
Jul 24 18:25:58.843834 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:58.843634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae8ab73-e781-4365-9fe4-c923d7753955-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf\" (UID: \"dae8ab73-e781-4365-9fe4-c923d7753955\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"
Jul 24 18:25:58.843834 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:58.843712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dae8ab73-e781-4365-9fe4-c923d7753955-maas-api-tls\") pod \"maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf\" (UID: \"dae8ab73-e781-4365-9fe4-c923d7753955\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"
Jul 24 18:25:58.858180 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:58.858109    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:58.944860 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:58.944821    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dae8ab73-e781-4365-9fe4-c923d7753955-maas-api-tls\") pod \"maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf\" (UID: \"dae8ab73-e781-4365-9fe4-c923d7753955\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"
Jul 24 18:25:58.945064 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:58.944971    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwgz7\" (UniqueName: \"kubernetes.io/projected/dae8ab73-e781-4365-9fe4-c923d7753955-kube-api-access-fwgz7\") pod \"maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf\" (UID: \"dae8ab73-e781-4365-9fe4-c923d7753955\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"
Jul 24 18:25:58.945064 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:58.944997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae8ab73-e781-4365-9fe4-c923d7753955-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf\" (UID: \"dae8ab73-e781-4365-9fe4-c923d7753955\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"
Jul 24 18:25:58.945637 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:58.945617    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae8ab73-e781-4365-9fe4-c923d7753955-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf\" (UID: \"dae8ab73-e781-4365-9fe4-c923d7753955\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"
Jul 24 18:25:58.947299 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:58.947275    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dae8ab73-e781-4365-9fe4-c923d7753955-maas-api-tls\") pod \"maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf\" (UID: \"dae8ab73-e781-4365-9fe4-c923d7753955\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"
Jul 24 18:25:58.956480 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:58.956455    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwgz7\" (UniqueName: \"kubernetes.io/projected/dae8ab73-e781-4365-9fe4-c923d7753955-kube-api-access-fwgz7\") pod \"maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf\" (UID: \"dae8ab73-e781-4365-9fe4-c923d7753955\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"
Jul 24 18:25:58.985527 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:58.985495    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"
Jul 24 18:25:59.131830 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.131796    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"]
Jul 24 18:25:59.131972 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:25:59.131947    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae8ab73_e781_4365_9fe4_c923d7753955.slice/crio-524adeecf4e4b0fe37f0c7de53f5d4b834aa3255daa34ef58d67ff28a26116f6 WatchSource:0}: Error finding container 524adeecf4e4b0fe37f0c7de53f5d4b834aa3255daa34ef58d67ff28a26116f6: Status 404 returned error can't find the container with id 524adeecf4e4b0fe37f0c7de53f5d4b834aa3255daa34ef58d67ff28a26116f6
Jul 24 18:25:59.185300 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.185270    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:59.268829 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.268798    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:59.272878 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.272851    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv"]
Jul 24 18:25:59.277426 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.277410    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv"
Jul 24 18:25:59.279936 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.279917    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-651f67\""
Jul 24 18:25:59.288243 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.288215    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv"]
Jul 24 18:25:59.449387 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.449306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7537ded-cf30-4e4a-bb72-11e41af2c87e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv\" (UID: \"b7537ded-cf30-4e4a-bb72-11e41af2c87e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv"
Jul 24 18:25:59.449387 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.449371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-864zm\" (UniqueName: \"kubernetes.io/projected/b7537ded-cf30-4e4a-bb72-11e41af2c87e-kube-api-access-864zm\") pod \"payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv\" (UID: \"b7537ded-cf30-4e4a-bb72-11e41af2c87e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv"
Jul 24 18:25:59.550367 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.550342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7537ded-cf30-4e4a-bb72-11e41af2c87e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv\" (UID: \"b7537ded-cf30-4e4a-bb72-11e41af2c87e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv"
Jul 24 18:25:59.550498 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.550395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-864zm\" (UniqueName: \"kubernetes.io/projected/b7537ded-cf30-4e4a-bb72-11e41af2c87e-kube-api-access-864zm\") pod \"payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv\" (UID: \"b7537ded-cf30-4e4a-bb72-11e41af2c87e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv"
Jul 24 18:25:59.551019 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.550999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7537ded-cf30-4e4a-bb72-11e41af2c87e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv\" (UID: \"b7537ded-cf30-4e4a-bb72-11e41af2c87e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv"
Jul 24 18:25:59.560235 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.560207    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-864zm\" (UniqueName: \"kubernetes.io/projected/b7537ded-cf30-4e4a-bb72-11e41af2c87e-kube-api-access-864zm\") pod \"payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv\" (UID: \"b7537ded-cf30-4e4a-bb72-11e41af2c87e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv"
Jul 24 18:25:59.590205 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.590178    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv"
Jul 24 18:25:59.743880 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.743851    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv"]
Jul 24 18:25:59.746018 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:25:59.745995    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7537ded_cf30_4e4a_bb72_11e41af2c87e.slice/crio-886bd51a6b83e55b05f1653d94bf6d24cc7898b1cd367a5090a0024d1c1deec9 WatchSource:0}: Error finding container 886bd51a6b83e55b05f1653d94bf6d24cc7898b1cd367a5090a0024d1c1deec9: Status 404 returned error can't find the container with id 886bd51a6b83e55b05f1653d94bf6d24cc7898b1cd367a5090a0024d1c1deec9
Jul 24 18:25:59.790854 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.790822    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf" event={"ID":"dae8ab73-e781-4365-9fe4-c923d7753955","Type":"ContainerStarted","Data":"5f3af909b2183fa42901798fae6624ce61e40a2e087b13de4f7b734db7e44e19"}
Jul 24 18:25:59.791022 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.790882    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf" event={"ID":"dae8ab73-e781-4365-9fe4-c923d7753955","Type":"ContainerStarted","Data":"524adeecf4e4b0fe37f0c7de53f5d4b834aa3255daa34ef58d67ff28a26116f6"}
Jul 24 18:25:59.791022 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.790916    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"
Jul 24 18:25:59.792082 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.792054    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv" event={"ID":"b7537ded-cf30-4e4a-bb72-11e41af2c87e","Type":"ContainerStarted","Data":"886bd51a6b83e55b05f1653d94bf6d24cc7898b1cd367a5090a0024d1c1deec9"}
Jul 24 18:25:59.810226 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:25:59.810182    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf" podStartSLOduration=1.483764772 podStartE2EDuration="1.810165392s" totalImagesPullingTime="326.40062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:25:59.808095975 +0000 UTC m=+2582.331454454" watchObservedRunningTime="2026-07-24 18:25:59.810165392 +0000 UTC m=+2582.333523870"
Jul 24 18:26:00.800587 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:26:00.800543    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv" event={"ID":"b7537ded-cf30-4e4a-bb72-11e41af2c87e","Type":"ContainerStarted","Data":"4be11fd27a1440eaf8eff0780e3635fb56ec700aa1271f7c8ab37d6e1d6fef6e"}
Jul 24 18:26:00.817215 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:26:00.817162    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv" podStartSLOduration=1.8171447120000002 podStartE2EDuration="1.817144712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:26:00.816435821 +0000 UTC m=+2583.339794302" watchObservedRunningTime="2026-07-24 18:26:00.817144712 +0000 UTC m=+2583.340503189"
Jul 24 18:26:01.806759 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:26:01.806718    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv"
Jul 24 18:26:03.091834 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:26:03.091752    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:26:05.806201 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:26:05.806172    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"
Jul 24 18:26:12.815215 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:26:12.815180    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv"
Jul 24 18:26:16.916498 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:26:16.916462    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:26:20.245684 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:26:20.245583    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9g7th_992231c7-b15d-4590-a3d5-615ed717052b/payload-processing/0.log"
Jul 24 18:26:22.751235 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:26:22.751201    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:26:57.045444 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:26:57.045414    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9g7th_992231c7-b15d-4590-a3d5-615ed717052b/payload-processing/0.log"
Jul 24 18:26:57.471279 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:26:57.471187    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:26:57.801566 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:26:57.801534    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:26:58.482158 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:26:58.482112    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:00.999498 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:00.999429    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:01.162763 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:01.162733    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"]
Jul 24 18:27:01.166596 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:01.166574    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"
Jul 24 18:27:01.170156 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:01.170126    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:01.171118 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:01.171098    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-1fd14b-serving-cert\""
Jul 24 18:27:01.189921 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:01.189884    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"]
Jul 24 18:27:01.261634 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:01.261542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8lmt\" (UniqueName: \"kubernetes.io/projected/3d89a33c-d410-4946-9a3e-6e30eb0786e7-kube-api-access-f8lmt\") pod \"maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm\" (UID: \"3d89a33c-d410-4946-9a3e-6e30eb0786e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"
Jul 24 18:27:01.261971 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:01.261939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d89a33c-d410-4946-9a3e-6e30eb0786e7-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm\" (UID: \"3d89a33c-d410-4946-9a3e-6e30eb0786e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"
Jul 24 18:27:01.262131 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:01.262109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d89a33c-d410-4946-9a3e-6e30eb0786e7-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm\" (UID: \"3d89a33c-d410-4946-9a3e-6e30eb0786e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"
Jul 24 18:27:01.294426 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:01.294388    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:01.363622 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:01.363585    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d89a33c-d410-4946-9a3e-6e30eb0786e7-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm\" (UID: \"3d89a33c-d410-4946-9a3e-6e30eb0786e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"
Jul 24 18:27:01.363859 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:01.363706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d89a33c-d410-4946-9a3e-6e30eb0786e7-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm\" (UID: \"3d89a33c-d410-4946-9a3e-6e30eb0786e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"
Jul 24 18:27:01.363859 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:01.363750    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8lmt\" (UniqueName: \"kubernetes.io/projected/3d89a33c-d410-4946-9a3e-6e30eb0786e7-kube-api-access-f8lmt\") pod \"maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm\" (UID: \"3d89a33c-d410-4946-9a3e-6e30eb0786e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"
Jul 24 18:27:01.364000 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:27:01.363893    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-serving-cert: secret "maas-api-e2e-ipp-cleanup-1fd14b-serving-cert" not found
Jul 24 18:27:01.364000 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:27:01.363982    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d89a33c-d410-4946-9a3e-6e30eb0786e7-maas-api-tls podName:3d89a33c-d410-4946-9a3e-6e30eb0786e7 nodeName:}" failed. No retries permitted until 2026-07-24 18:27:01.863960229 +0000 UTC m=+2644.387318709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d89a33c-d410-4946-9a3e-6e30eb0786e7-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm" (UID: "3d89a33c-d410-4946-9a3e-6e30eb0786e7") : secret "maas-api-e2e-ipp-cleanup-1fd14b-serving-cert" not found
Jul 24 18:27:01.364409 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:01.364389    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d89a33c-d410-4946-9a3e-6e30eb0786e7-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm\" (UID: \"3d89a33c-d410-4946-9a3e-6e30eb0786e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"
Jul 24 18:27:01.384677 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:01.384639    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8lmt\" (UniqueName: \"kubernetes.io/projected/3d89a33c-d410-4946-9a3e-6e30eb0786e7-kube-api-access-f8lmt\") pod \"maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm\" (UID: \"3d89a33c-d410-4946-9a3e-6e30eb0786e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"
Jul 24 18:27:01.624368 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:01.624338    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:01.868933 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:01.868869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d89a33c-d410-4946-9a3e-6e30eb0786e7-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm\" (UID: \"3d89a33c-d410-4946-9a3e-6e30eb0786e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"
Jul 24 18:27:01.871461 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:01.871433    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d89a33c-d410-4946-9a3e-6e30eb0786e7-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm\" (UID: \"3d89a33c-d410-4946-9a3e-6e30eb0786e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"
Jul 24 18:27:02.078036 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:02.077996    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"
Jul 24 18:27:02.219709 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:02.219675    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:02.251145 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:02.251116    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"]
Jul 24 18:27:02.254110 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:27:02.254070    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d89a33c_d410_4946_9a3e_6e30eb0786e7.slice/crio-dd56394cd3615eff4ab0330035c319e7eef20b44b8256e1c0ec0cc917a3aa5e0 WatchSource:0}: Error finding container dd56394cd3615eff4ab0330035c319e7eef20b44b8256e1c0ec0cc917a3aa5e0: Status 404 returned error can't find the container with id dd56394cd3615eff4ab0330035c319e7eef20b44b8256e1c0ec0cc917a3aa5e0
Jul 24 18:27:02.436069 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:02.435985    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:03.134603 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:03.134555    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm" event={"ID":"3d89a33c-d410-4946-9a3e-6e30eb0786e7","Type":"ContainerStarted","Data":"207e54677502df111d5086674c46c09cf6f221438a672e3ccdb6bd1f95b8e7c1"}
Jul 24 18:27:03.135179 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:03.134625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm" event={"ID":"3d89a33c-d410-4946-9a3e-6e30eb0786e7","Type":"ContainerStarted","Data":"dd56394cd3615eff4ab0330035c319e7eef20b44b8256e1c0ec0cc917a3aa5e0"}
Jul 24 18:27:03.135179 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:03.134653    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"
Jul 24 18:27:03.173083 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:03.173028    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm" podStartSLOduration=1.778765419 podStartE2EDuration="2.17301021s" totalImagesPullingTime="394.244791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:27:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:27:03.17236297 +0000 UTC m=+2645.695721448" watchObservedRunningTime="2026-07-24 18:27:03.17301021 +0000 UTC m=+2645.696368690"
Jul 24 18:27:05.486503 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:05.486466    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:09.145617 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:09.145585    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"
Jul 24 18:27:27.057827 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:27.057764    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"]
Jul 24 18:27:27.058324 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:27.058051    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm" podUID="3d89a33c-d410-4946-9a3e-6e30eb0786e7" containerName="maas-api" containerID="cri-o://207e54677502df111d5086674c46c09cf6f221438a672e3ccdb6bd1f95b8e7c1" gracePeriod=30
Jul 24 18:27:27.090378 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:27.090350    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:27.193161 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:27.193130    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:27.273260 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:27.273228    2578 generic.go:356] "Generic (PLEG): container finished" podID="3d89a33c-d410-4946-9a3e-6e30eb0786e7" containerID="207e54677502df111d5086674c46c09cf6f221438a672e3ccdb6bd1f95b8e7c1" exitCode=0
Jul 24 18:27:27.273423 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:27.273276    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm" event={"ID":"3d89a33c-d410-4946-9a3e-6e30eb0786e7","Type":"ContainerDied","Data":"207e54677502df111d5086674c46c09cf6f221438a672e3ccdb6bd1f95b8e7c1"}
Jul 24 18:27:27.291881 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:27.291838    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:27.322396 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:27.322334    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"
Jul 24 18:27:27.425048 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:27.425015    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8lmt\" (UniqueName: \"kubernetes.io/projected/3d89a33c-d410-4946-9a3e-6e30eb0786e7-kube-api-access-f8lmt\") pod \"3d89a33c-d410-4946-9a3e-6e30eb0786e7\" (UID: \"3d89a33c-d410-4946-9a3e-6e30eb0786e7\") "
Jul 24 18:27:27.425243 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:27.425108    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d89a33c-d410-4946-9a3e-6e30eb0786e7-openshift-service-ca\") pod \"3d89a33c-d410-4946-9a3e-6e30eb0786e7\" (UID: \"3d89a33c-d410-4946-9a3e-6e30eb0786e7\") "
Jul 24 18:27:27.425243 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:27.425144    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d89a33c-d410-4946-9a3e-6e30eb0786e7-maas-api-tls\") pod \"3d89a33c-d410-4946-9a3e-6e30eb0786e7\" (UID: \"3d89a33c-d410-4946-9a3e-6e30eb0786e7\") "
Jul 24 18:27:27.425524 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:27.425495    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d89a33c-d410-4946-9a3e-6e30eb0786e7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3d89a33c-d410-4946-9a3e-6e30eb0786e7" (UID: "3d89a33c-d410-4946-9a3e-6e30eb0786e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:27:27.427297 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:27.427270    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d89a33c-d410-4946-9a3e-6e30eb0786e7-kube-api-access-f8lmt" (OuterVolumeSpecName: "kube-api-access-f8lmt") pod "3d89a33c-d410-4946-9a3e-6e30eb0786e7" (UID: "3d89a33c-d410-4946-9a3e-6e30eb0786e7"). InnerVolumeSpecName "kube-api-access-f8lmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:27:27.427415 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:27.427301    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d89a33c-d410-4946-9a3e-6e30eb0786e7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3d89a33c-d410-4946-9a3e-6e30eb0786e7" (UID: "3d89a33c-d410-4946-9a3e-6e30eb0786e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:27:27.526320 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:27.526287    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d89a33c-d410-4946-9a3e-6e30eb0786e7-openshift-service-ca\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:27:27.526320 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:27.526316    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d89a33c-d410-4946-9a3e-6e30eb0786e7-maas-api-tls\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:27:27.526320 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:27.526326    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8lmt\" (UniqueName: \"kubernetes.io/projected/3d89a33c-d410-4946-9a3e-6e30eb0786e7-kube-api-access-f8lmt\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:27:28.280028 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:28.279952    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"
Jul 24 18:27:28.280441 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:28.279954    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm" event={"ID":"3d89a33c-d410-4946-9a3e-6e30eb0786e7","Type":"ContainerDied","Data":"dd56394cd3615eff4ab0330035c319e7eef20b44b8256e1c0ec0cc917a3aa5e0"}
Jul 24 18:27:28.280441 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:28.280087    2578 scope.go:117] "RemoveContainer" containerID="207e54677502df111d5086674c46c09cf6f221438a672e3ccdb6bd1f95b8e7c1"
Jul 24 18:27:28.298142 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:28.298112    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"]
Jul 24 18:27:28.302977 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:28.302949    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1fd14b-d95fccf8-dwzrm"]
Jul 24 18:27:29.967596 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:29.967558    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d89a33c-d410-4946-9a3e-6e30eb0786e7" path="/var/lib/kubelet/pods/3d89a33c-d410-4946-9a3e-6e30eb0786e7/volumes"
Jul 24 18:27:31.591537 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:31.591497    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:31.998513 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:31.998427    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:37.667523 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:37.667490    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:37.733395 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:37.733366    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:37.975859 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:37.975734    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:38.259631 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:38.259552    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:57.517675 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:57.517584    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"]
Jul 24 18:27:57.518151 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:57.517881    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv" podUID="deb6fb89-9bca-4788-9807-f70d4dc32b2b" containerName="maas-api" containerID="cri-o://73bef9d5cb82de8675ffc75131ad1cfe59d87309235b0a42ec6369b03926e61d" gracePeriod=30
Jul 24 18:27:57.550142 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:57.550110    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:57.582461 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:57.582418    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr"]
Jul 24 18:27:57.582724 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:57.582676    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr" podUID="21231bc0-059e-4466-b88b-a7eac3530a56" containerName="payload-pre-processing" containerID="cri-o://7ab8d908f6049f872e6be0356296b4e35462d0d218abb40b78171732d12688bb" gracePeriod=30
Jul 24 18:27:57.649282 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:57.649242    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:57.696531 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:57.696492    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv" podUID="deb6fb89-9bca-4788-9807-f70d4dc32b2b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.76:8443/health\": dial tcp 10.133.0.76:8443: connect: connection refused"
Jul 24 18:27:57.748591 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:57.748554    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:57.885339 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:57.885314    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"
Jul 24 18:27:57.888362 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:57.888344    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr"
Jul 24 18:27:58.016197 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.016159    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f66rp\" (UniqueName: \"kubernetes.io/projected/21231bc0-059e-4466-b88b-a7eac3530a56-kube-api-access-f66rp\") pod \"21231bc0-059e-4466-b88b-a7eac3530a56\" (UID: \"21231bc0-059e-4466-b88b-a7eac3530a56\") "
Jul 24 18:27:58.016383 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.016243    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21231bc0-059e-4466-b88b-a7eac3530a56-plugins-config-volume\") pod \"21231bc0-059e-4466-b88b-a7eac3530a56\" (UID: \"21231bc0-059e-4466-b88b-a7eac3530a56\") "
Jul 24 18:27:58.016383 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.016298    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb6fb89-9bca-4788-9807-f70d4dc32b2b-maas-api-tls\") pod \"deb6fb89-9bca-4788-9807-f70d4dc32b2b\" (UID: \"deb6fb89-9bca-4788-9807-f70d4dc32b2b\") "
Jul 24 18:27:58.016383 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.016340    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mhhj\" (UniqueName: \"kubernetes.io/projected/deb6fb89-9bca-4788-9807-f70d4dc32b2b-kube-api-access-9mhhj\") pod \"deb6fb89-9bca-4788-9807-f70d4dc32b2b\" (UID: \"deb6fb89-9bca-4788-9807-f70d4dc32b2b\") "
Jul 24 18:27:58.016383 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.016371    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb6fb89-9bca-4788-9807-f70d4dc32b2b-openshift-service-ca\") pod \"deb6fb89-9bca-4788-9807-f70d4dc32b2b\" (UID: \"deb6fb89-9bca-4788-9807-f70d4dc32b2b\") "
Jul 24 18:27:58.016666 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.016633    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21231bc0-059e-4466-b88b-a7eac3530a56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "21231bc0-059e-4466-b88b-a7eac3530a56" (UID: "21231bc0-059e-4466-b88b-a7eac3530a56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:27:58.016878 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.016851    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb6fb89-9bca-4788-9807-f70d4dc32b2b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "deb6fb89-9bca-4788-9807-f70d4dc32b2b" (UID: "deb6fb89-9bca-4788-9807-f70d4dc32b2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:27:58.018584 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.018553    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21231bc0-059e-4466-b88b-a7eac3530a56-kube-api-access-f66rp" (OuterVolumeSpecName: "kube-api-access-f66rp") pod "21231bc0-059e-4466-b88b-a7eac3530a56" (UID: "21231bc0-059e-4466-b88b-a7eac3530a56"). InnerVolumeSpecName "kube-api-access-f66rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:27:58.018697 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.018611    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb6fb89-9bca-4788-9807-f70d4dc32b2b-kube-api-access-9mhhj" (OuterVolumeSpecName: "kube-api-access-9mhhj") pod "deb6fb89-9bca-4788-9807-f70d4dc32b2b" (UID: "deb6fb89-9bca-4788-9807-f70d4dc32b2b"). InnerVolumeSpecName "kube-api-access-9mhhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:27:58.018697 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.018650    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb6fb89-9bca-4788-9807-f70d4dc32b2b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "deb6fb89-9bca-4788-9807-f70d4dc32b2b" (UID: "deb6fb89-9bca-4788-9807-f70d4dc32b2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:27:58.117433 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.117350    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb6fb89-9bca-4788-9807-f70d4dc32b2b-openshift-service-ca\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:27:58.117433 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.117376    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f66rp\" (UniqueName: \"kubernetes.io/projected/21231bc0-059e-4466-b88b-a7eac3530a56-kube-api-access-f66rp\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:27:58.117433 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.117386    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21231bc0-059e-4466-b88b-a7eac3530a56-plugins-config-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:27:58.117433 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.117395    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb6fb89-9bca-4788-9807-f70d4dc32b2b-maas-api-tls\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:27:58.117433 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.117403    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mhhj\" (UniqueName: \"kubernetes.io/projected/deb6fb89-9bca-4788-9807-f70d4dc32b2b-kube-api-access-9mhhj\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:27:58.449359 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.449262    2578 generic.go:356] "Generic (PLEG): container finished" podID="21231bc0-059e-4466-b88b-a7eac3530a56" containerID="7ab8d908f6049f872e6be0356296b4e35462d0d218abb40b78171732d12688bb" exitCode=0
Jul 24 18:27:58.449359 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.449307    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr"
Jul 24 18:27:58.449359 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.449344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr" event={"ID":"21231bc0-059e-4466-b88b-a7eac3530a56","Type":"ContainerDied","Data":"7ab8d908f6049f872e6be0356296b4e35462d0d218abb40b78171732d12688bb"}
Jul 24 18:27:58.449632 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.449404    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr" event={"ID":"21231bc0-059e-4466-b88b-a7eac3530a56","Type":"ContainerDied","Data":"e75d517bec708c954aae299cc850f535826a605be7622cb8010e17c6093905e8"}
Jul 24 18:27:58.449632 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.449439    2578 scope.go:117] "RemoveContainer" containerID="7ab8d908f6049f872e6be0356296b4e35462d0d218abb40b78171732d12688bb"
Jul 24 18:27:58.450688 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.450666    2578 generic.go:356] "Generic (PLEG): container finished" podID="deb6fb89-9bca-4788-9807-f70d4dc32b2b" containerID="73bef9d5cb82de8675ffc75131ad1cfe59d87309235b0a42ec6369b03926e61d" exitCode=0
Jul 24 18:27:58.450833 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.450691    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv" event={"ID":"deb6fb89-9bca-4788-9807-f70d4dc32b2b","Type":"ContainerDied","Data":"73bef9d5cb82de8675ffc75131ad1cfe59d87309235b0a42ec6369b03926e61d"}
Jul 24 18:27:58.450833 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.450729    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"
Jul 24 18:27:58.450833 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.450740    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv" event={"ID":"deb6fb89-9bca-4788-9807-f70d4dc32b2b","Type":"ContainerDied","Data":"3b1f46714a7f2e28a6cd4d190885b7f4d130e49aaca53e02b93ec6776e84d07b"}
Jul 24 18:27:58.460335 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.460312    2578 scope.go:117] "RemoveContainer" containerID="7ab8d908f6049f872e6be0356296b4e35462d0d218abb40b78171732d12688bb"
Jul 24 18:27:58.461522 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:27:58.461364    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ab8d908f6049f872e6be0356296b4e35462d0d218abb40b78171732d12688bb\": container with ID starting with 7ab8d908f6049f872e6be0356296b4e35462d0d218abb40b78171732d12688bb not found: ID does not exist" containerID="7ab8d908f6049f872e6be0356296b4e35462d0d218abb40b78171732d12688bb"
Jul 24 18:27:58.461522 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.461400    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab8d908f6049f872e6be0356296b4e35462d0d218abb40b78171732d12688bb"} err="failed to get container status \"7ab8d908f6049f872e6be0356296b4e35462d0d218abb40b78171732d12688bb\": rpc error: code = NotFound desc = could not find container \"7ab8d908f6049f872e6be0356296b4e35462d0d218abb40b78171732d12688bb\": container with ID starting with 7ab8d908f6049f872e6be0356296b4e35462d0d218abb40b78171732d12688bb not found: ID does not exist"
Jul 24 18:27:58.461522 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.461423    2578 scope.go:117] "RemoveContainer" containerID="73bef9d5cb82de8675ffc75131ad1cfe59d87309235b0a42ec6369b03926e61d"
Jul 24 18:27:58.472945 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.472919    2578 scope.go:117] "RemoveContainer" containerID="73bef9d5cb82de8675ffc75131ad1cfe59d87309235b0a42ec6369b03926e61d"
Jul 24 18:27:58.473191 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:27:58.473172    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73bef9d5cb82de8675ffc75131ad1cfe59d87309235b0a42ec6369b03926e61d\": container with ID starting with 73bef9d5cb82de8675ffc75131ad1cfe59d87309235b0a42ec6369b03926e61d not found: ID does not exist" containerID="73bef9d5cb82de8675ffc75131ad1cfe59d87309235b0a42ec6369b03926e61d"
Jul 24 18:27:58.473235 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.473199    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73bef9d5cb82de8675ffc75131ad1cfe59d87309235b0a42ec6369b03926e61d"} err="failed to get container status \"73bef9d5cb82de8675ffc75131ad1cfe59d87309235b0a42ec6369b03926e61d\": rpc error: code = NotFound desc = could not find container \"73bef9d5cb82de8675ffc75131ad1cfe59d87309235b0a42ec6369b03926e61d\": container with ID starting with 73bef9d5cb82de8675ffc75131ad1cfe59d87309235b0a42ec6369b03926e61d not found: ID does not exist"
Jul 24 18:27:58.479016 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.478991    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"]
Jul 24 18:27:58.481020 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.481000    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d91e94-5bff6bf4b5-mrzdv"]
Jul 24 18:27:58.491544 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.491520    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr"]
Jul 24 18:27:58.496066 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:58.496043    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d91e94-7477689fb-vggwr"]
Jul 24 18:27:59.967864 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:59.967823    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21231bc0-059e-4466-b88b-a7eac3530a56" path="/var/lib/kubelet/pods/21231bc0-059e-4466-b88b-a7eac3530a56/volumes"
Jul 24 18:27:59.968433 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:27:59.968412    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deb6fb89-9bca-4788-9807-f70d4dc32b2b" path="/var/lib/kubelet/pods/deb6fb89-9bca-4788-9807-f70d4dc32b2b/volumes"
Jul 24 18:28:02.147556 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:28:02.147504    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:02.562296 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:28:02.562265    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:07.635847 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:28:07.635820    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:44.483608 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:28:44.483574    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:44.822157 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:28:44.822124    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:50.378172 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:28:50.378132    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:50.453384 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:28:50.453351    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:50.662541 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:28:50.662449    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:50.877080 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:28:50.877045    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:01.746686 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:01.746641    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"]
Jul 24 18:29:01.747157 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:01.746900    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf" podUID="dae8ab73-e781-4365-9fe4-c923d7753955" containerName="maas-api" containerID="cri-o://5f3af909b2183fa42901798fae6624ce61e40a2e087b13de4f7b734db7e44e19" gracePeriod=30
Jul 24 18:29:01.777820 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:01.777765    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:01.858723 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:01.858688    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv"]
Jul 24 18:29:01.859027 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:01.858997    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv" podUID="b7537ded-cf30-4e4a-bb72-11e41af2c87e" containerName="payload-pre-processing" containerID="cri-o://4be11fd27a1440eaf8eff0780e3635fb56ec700aa1271f7c8ab37d6e1d6fef6e" gracePeriod=30
Jul 24 18:29:01.903158 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:01.903122    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:02.007198 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.007100    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:02.037872 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.037834    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"
Jul 24 18:29:02.125495 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.125465    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dae8ab73-e781-4365-9fe4-c923d7753955-maas-api-tls\") pod \"dae8ab73-e781-4365-9fe4-c923d7753955\" (UID: \"dae8ab73-e781-4365-9fe4-c923d7753955\") "
Jul 24 18:29:02.125495 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.125504    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae8ab73-e781-4365-9fe4-c923d7753955-openshift-service-ca\") pod \"dae8ab73-e781-4365-9fe4-c923d7753955\" (UID: \"dae8ab73-e781-4365-9fe4-c923d7753955\") "
Jul 24 18:29:02.125766 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.125535    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwgz7\" (UniqueName: \"kubernetes.io/projected/dae8ab73-e781-4365-9fe4-c923d7753955-kube-api-access-fwgz7\") pod \"dae8ab73-e781-4365-9fe4-c923d7753955\" (UID: \"dae8ab73-e781-4365-9fe4-c923d7753955\") "
Jul 24 18:29:02.125925 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.125901    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae8ab73-e781-4365-9fe4-c923d7753955-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dae8ab73-e781-4365-9fe4-c923d7753955" (UID: "dae8ab73-e781-4365-9fe4-c923d7753955"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:29:02.127864 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.127827    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae8ab73-e781-4365-9fe4-c923d7753955-kube-api-access-fwgz7" (OuterVolumeSpecName: "kube-api-access-fwgz7") pod "dae8ab73-e781-4365-9fe4-c923d7753955" (UID: "dae8ab73-e781-4365-9fe4-c923d7753955"). InnerVolumeSpecName "kube-api-access-fwgz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:29:02.127981 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.127912    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dae8ab73-e781-4365-9fe4-c923d7753955-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dae8ab73-e781-4365-9fe4-c923d7753955" (UID: "dae8ab73-e781-4365-9fe4-c923d7753955"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:29:02.152903 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.152881    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv"
Jul 24 18:29:02.226581 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.226542    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-864zm\" (UniqueName: \"kubernetes.io/projected/b7537ded-cf30-4e4a-bb72-11e41af2c87e-kube-api-access-864zm\") pod \"b7537ded-cf30-4e4a-bb72-11e41af2c87e\" (UID: \"b7537ded-cf30-4e4a-bb72-11e41af2c87e\") "
Jul 24 18:29:02.226761 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.226635    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7537ded-cf30-4e4a-bb72-11e41af2c87e-plugins-config-volume\") pod \"b7537ded-cf30-4e4a-bb72-11e41af2c87e\" (UID: \"b7537ded-cf30-4e4a-bb72-11e41af2c87e\") "
Jul 24 18:29:02.226848 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.226817    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dae8ab73-e781-4365-9fe4-c923d7753955-maas-api-tls\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:29:02.226848 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.226831    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae8ab73-e781-4365-9fe4-c923d7753955-openshift-service-ca\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:29:02.226848 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.226841    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwgz7\" (UniqueName: \"kubernetes.io/projected/dae8ab73-e781-4365-9fe4-c923d7753955-kube-api-access-fwgz7\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:29:02.227012 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.226989    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7537ded-cf30-4e4a-bb72-11e41af2c87e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7537ded-cf30-4e4a-bb72-11e41af2c87e" (UID: "b7537ded-cf30-4e4a-bb72-11e41af2c87e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:29:02.228671 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.228645    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7537ded-cf30-4e4a-bb72-11e41af2c87e-kube-api-access-864zm" (OuterVolumeSpecName: "kube-api-access-864zm") pod "b7537ded-cf30-4e4a-bb72-11e41af2c87e" (UID: "b7537ded-cf30-4e4a-bb72-11e41af2c87e"). InnerVolumeSpecName "kube-api-access-864zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:29:02.327965 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.327920    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7537ded-cf30-4e4a-bb72-11e41af2c87e-plugins-config-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:29:02.327965 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.327953    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-864zm\" (UniqueName: \"kubernetes.io/projected/b7537ded-cf30-4e4a-bb72-11e41af2c87e-kube-api-access-864zm\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:29:02.794564 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.794524    2578 generic.go:356] "Generic (PLEG): container finished" podID="b7537ded-cf30-4e4a-bb72-11e41af2c87e" containerID="4be11fd27a1440eaf8eff0780e3635fb56ec700aa1271f7c8ab37d6e1d6fef6e" exitCode=0
Jul 24 18:29:02.795071 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.794570    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv"
Jul 24 18:29:02.795071 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.794611    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv" event={"ID":"b7537ded-cf30-4e4a-bb72-11e41af2c87e","Type":"ContainerDied","Data":"4be11fd27a1440eaf8eff0780e3635fb56ec700aa1271f7c8ab37d6e1d6fef6e"}
Jul 24 18:29:02.795071 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.794667    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv" event={"ID":"b7537ded-cf30-4e4a-bb72-11e41af2c87e","Type":"ContainerDied","Data":"886bd51a6b83e55b05f1653d94bf6d24cc7898b1cd367a5090a0024d1c1deec9"}
Jul 24 18:29:02.795071 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.794699    2578 scope.go:117] "RemoveContainer" containerID="4be11fd27a1440eaf8eff0780e3635fb56ec700aa1271f7c8ab37d6e1d6fef6e"
Jul 24 18:29:02.796318 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.796274    2578 generic.go:356] "Generic (PLEG): container finished" podID="dae8ab73-e781-4365-9fe4-c923d7753955" containerID="5f3af909b2183fa42901798fae6624ce61e40a2e087b13de4f7b734db7e44e19" exitCode=0
Jul 24 18:29:02.796425 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.796337    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"
Jul 24 18:29:02.796425 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.796390    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf" event={"ID":"dae8ab73-e781-4365-9fe4-c923d7753955","Type":"ContainerDied","Data":"5f3af909b2183fa42901798fae6624ce61e40a2e087b13de4f7b734db7e44e19"}
Jul 24 18:29:02.796528 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.796445    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf" event={"ID":"dae8ab73-e781-4365-9fe4-c923d7753955","Type":"ContainerDied","Data":"524adeecf4e4b0fe37f0c7de53f5d4b834aa3255daa34ef58d67ff28a26116f6"}
Jul 24 18:29:02.804459 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.804441    2578 scope.go:117] "RemoveContainer" containerID="4be11fd27a1440eaf8eff0780e3635fb56ec700aa1271f7c8ab37d6e1d6fef6e"
Jul 24 18:29:02.804835 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:29:02.804803    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be11fd27a1440eaf8eff0780e3635fb56ec700aa1271f7c8ab37d6e1d6fef6e\": container with ID starting with 4be11fd27a1440eaf8eff0780e3635fb56ec700aa1271f7c8ab37d6e1d6fef6e not found: ID does not exist" containerID="4be11fd27a1440eaf8eff0780e3635fb56ec700aa1271f7c8ab37d6e1d6fef6e"
Jul 24 18:29:02.804936 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.804847    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be11fd27a1440eaf8eff0780e3635fb56ec700aa1271f7c8ab37d6e1d6fef6e"} err="failed to get container status \"4be11fd27a1440eaf8eff0780e3635fb56ec700aa1271f7c8ab37d6e1d6fef6e\": rpc error: code = NotFound desc = could not find container \"4be11fd27a1440eaf8eff0780e3635fb56ec700aa1271f7c8ab37d6e1d6fef6e\": container with ID starting with 4be11fd27a1440eaf8eff0780e3635fb56ec700aa1271f7c8ab37d6e1d6fef6e not found: ID does not exist"
Jul 24 18:29:02.804936 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.804871    2578 scope.go:117] "RemoveContainer" containerID="5f3af909b2183fa42901798fae6624ce61e40a2e087b13de4f7b734db7e44e19"
Jul 24 18:29:02.816804 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.816759    2578 scope.go:117] "RemoveContainer" containerID="5f3af909b2183fa42901798fae6624ce61e40a2e087b13de4f7b734db7e44e19"
Jul 24 18:29:02.817302 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:29:02.817280    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f3af909b2183fa42901798fae6624ce61e40a2e087b13de4f7b734db7e44e19\": container with ID starting with 5f3af909b2183fa42901798fae6624ce61e40a2e087b13de4f7b734db7e44e19 not found: ID does not exist" containerID="5f3af909b2183fa42901798fae6624ce61e40a2e087b13de4f7b734db7e44e19"
Jul 24 18:29:02.817402 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.817312    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3af909b2183fa42901798fae6624ce61e40a2e087b13de4f7b734db7e44e19"} err="failed to get container status \"5f3af909b2183fa42901798fae6624ce61e40a2e087b13de4f7b734db7e44e19\": rpc error: code = NotFound desc = could not find container \"5f3af909b2183fa42901798fae6624ce61e40a2e087b13de4f7b734db7e44e19\": container with ID starting with 5f3af909b2183fa42901798fae6624ce61e40a2e087b13de4f7b734db7e44e19 not found: ID does not exist"
Jul 24 18:29:02.823933 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.823912    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"]
Jul 24 18:29:02.826074 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.826052    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-651f67-69d489f475-f5dgf"]
Jul 24 18:29:02.837532 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.837509    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv"]
Jul 24 18:29:02.845702 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:02.845681    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-651f67-6987849cbd-8qdbv"]
Jul 24 18:29:03.968318 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:03.968289    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7537ded-cf30-4e4a-bb72-11e41af2c87e" path="/var/lib/kubelet/pods/b7537ded-cf30-4e4a-bb72-11e41af2c87e/volumes"
Jul 24 18:29:03.968699 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:03.968690    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dae8ab73-e781-4365-9fe4-c923d7753955" path="/var/lib/kubelet/pods/dae8ab73-e781-4365-9fe4-c923d7753955/volumes"
Jul 24 18:29:04.686216 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:04.686183    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:04.750077 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:04.750045    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:06.080970 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:06.080931    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:06.473304 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:06.473222    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:10.889508 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:10.889458    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:10.971971 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:10.971934    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:12.412164 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:12.412134    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:12.483484 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:12.483451    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:12.691292 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:12.691216    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:12.927056 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:12.927021    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:23.518904 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.518866    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"]
Jul 24 18:29:23.521908 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.519165    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l" podUID="ca089800-66f3-482f-92db-917bc1868fb0" containerName="maas-api" containerID="cri-o://8b14872417568ebe76b21c64f63891ea357d552a250081bd2ddbcef4c43ae87e" gracePeriod=30
Jul 24 18:29:23.553357 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.553316    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:23.579625 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.579589    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84"]
Jul 24 18:29:23.579900 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.579870    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84" podUID="fa0c01c6-ed03-498b-a386-44c85d71e601" containerName="payload-processing" containerID="cri-o://0c4a31768040ba61b345f074ac7402971d701519132e809d5061411c0ffdff03" gracePeriod=30
Jul 24 18:29:23.588363 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.588335    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm"]
Jul 24 18:29:23.588536 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.588518    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm" podUID="846247f7-38c6-438b-b04b-8681967a7124" containerName="payload-pre-processing" containerID="cri-o://759ff495741ab57757ced944fc7df40d8f6fe164c95648be72cf241f44861754" gracePeriod=30
Jul 24 18:29:23.627028 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.626998    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:23.695357 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.695328    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:23.862979 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.862947    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84"
Jul 24 18:29:23.901217 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.901190    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"
Jul 24 18:29:23.908182 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.908148    2578 generic.go:356] "Generic (PLEG): container finished" podID="ca089800-66f3-482f-92db-917bc1868fb0" containerID="8b14872417568ebe76b21c64f63891ea357d552a250081bd2ddbcef4c43ae87e" exitCode=0
Jul 24 18:29:23.908318 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.908188    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"
Jul 24 18:29:23.908318 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.908260    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l" event={"ID":"ca089800-66f3-482f-92db-917bc1868fb0","Type":"ContainerDied","Data":"8b14872417568ebe76b21c64f63891ea357d552a250081bd2ddbcef4c43ae87e"}
Jul 24 18:29:23.908318 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.908304    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l" event={"ID":"ca089800-66f3-482f-92db-917bc1868fb0","Type":"ContainerDied","Data":"6db1f673200f64b09d02883dc8d8729d31d0af205ff9e02ad21eabc2898d05b2"}
Jul 24 18:29:23.908503 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.908334    2578 scope.go:117] "RemoveContainer" containerID="8b14872417568ebe76b21c64f63891ea357d552a250081bd2ddbcef4c43ae87e"
Jul 24 18:29:23.909673 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.909649    2578 generic.go:356] "Generic (PLEG): container finished" podID="fa0c01c6-ed03-498b-a386-44c85d71e601" containerID="0c4a31768040ba61b345f074ac7402971d701519132e809d5061411c0ffdff03" exitCode=0
Jul 24 18:29:23.909822 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.909696    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84"
Jul 24 18:29:23.909822 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.909813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84" event={"ID":"fa0c01c6-ed03-498b-a386-44c85d71e601","Type":"ContainerDied","Data":"0c4a31768040ba61b345f074ac7402971d701519132e809d5061411c0ffdff03"}
Jul 24 18:29:23.909944 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.909854    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84" event={"ID":"fa0c01c6-ed03-498b-a386-44c85d71e601","Type":"ContainerDied","Data":"720b4eab83dac4a1b5099c5830ae70b673d22000c4438c70e579952260ba57b8"}
Jul 24 18:29:23.911362 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.911340    2578 generic.go:356] "Generic (PLEG): container finished" podID="846247f7-38c6-438b-b04b-8681967a7124" containerID="759ff495741ab57757ced944fc7df40d8f6fe164c95648be72cf241f44861754" exitCode=0
Jul 24 18:29:23.911485 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.911406    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm" event={"ID":"846247f7-38c6-438b-b04b-8681967a7124","Type":"ContainerDied","Data":"759ff495741ab57757ced944fc7df40d8f6fe164c95648be72cf241f44861754"}
Jul 24 18:29:23.913584 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.913565    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm"
Jul 24 18:29:23.924573 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.923244    2578 scope.go:117] "RemoveContainer" containerID="8b14872417568ebe76b21c64f63891ea357d552a250081bd2ddbcef4c43ae87e"
Jul 24 18:29:23.924573 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:29:23.924236    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b14872417568ebe76b21c64f63891ea357d552a250081bd2ddbcef4c43ae87e\": container with ID starting with 8b14872417568ebe76b21c64f63891ea357d552a250081bd2ddbcef4c43ae87e not found: ID does not exist" containerID="8b14872417568ebe76b21c64f63891ea357d552a250081bd2ddbcef4c43ae87e"
Jul 24 18:29:23.924573 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.924272    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b14872417568ebe76b21c64f63891ea357d552a250081bd2ddbcef4c43ae87e"} err="failed to get container status \"8b14872417568ebe76b21c64f63891ea357d552a250081bd2ddbcef4c43ae87e\": rpc error: code = NotFound desc = could not find container \"8b14872417568ebe76b21c64f63891ea357d552a250081bd2ddbcef4c43ae87e\": container with ID starting with 8b14872417568ebe76b21c64f63891ea357d552a250081bd2ddbcef4c43ae87e not found: ID does not exist"
Jul 24 18:29:23.924573 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.924300    2578 scope.go:117] "RemoveContainer" containerID="0c4a31768040ba61b345f074ac7402971d701519132e809d5061411c0ffdff03"
Jul 24 18:29:23.934544 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.934526    2578 scope.go:117] "RemoveContainer" containerID="0c4a31768040ba61b345f074ac7402971d701519132e809d5061411c0ffdff03"
Jul 24 18:29:23.934758 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.934739    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0c01c6-ed03-498b-a386-44c85d71e601-plugins-config-volume\") pod \"fa0c01c6-ed03-498b-a386-44c85d71e601\" (UID: \"fa0c01c6-ed03-498b-a386-44c85d71e601\") "
Jul 24 18:29:23.934859 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:29:23.934766    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4a31768040ba61b345f074ac7402971d701519132e809d5061411c0ffdff03\": container with ID starting with 0c4a31768040ba61b345f074ac7402971d701519132e809d5061411c0ffdff03 not found: ID does not exist" containerID="0c4a31768040ba61b345f074ac7402971d701519132e809d5061411c0ffdff03"
Jul 24 18:29:23.934859 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.934810    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zj9b\" (UniqueName: \"kubernetes.io/projected/fa0c01c6-ed03-498b-a386-44c85d71e601-kube-api-access-2zj9b\") pod \"fa0c01c6-ed03-498b-a386-44c85d71e601\" (UID: \"fa0c01c6-ed03-498b-a386-44c85d71e601\") "
Jul 24 18:29:23.934859 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.934820    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4a31768040ba61b345f074ac7402971d701519132e809d5061411c0ffdff03"} err="failed to get container status \"0c4a31768040ba61b345f074ac7402971d701519132e809d5061411c0ffdff03\": rpc error: code = NotFound desc = could not find container \"0c4a31768040ba61b345f074ac7402971d701519132e809d5061411c0ffdff03\": container with ID starting with 0c4a31768040ba61b345f074ac7402971d701519132e809d5061411c0ffdff03 not found: ID does not exist"
Jul 24 18:29:23.935067 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.935047    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0c01c6-ed03-498b-a386-44c85d71e601-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa0c01c6-ed03-498b-a386-44c85d71e601" (UID: "fa0c01c6-ed03-498b-a386-44c85d71e601"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:29:23.935184 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.935167    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0c01c6-ed03-498b-a386-44c85d71e601-plugins-config-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:29:23.936790 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:23.936755    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0c01c6-ed03-498b-a386-44c85d71e601-kube-api-access-2zj9b" (OuterVolumeSpecName: "kube-api-access-2zj9b") pod "fa0c01c6-ed03-498b-a386-44c85d71e601" (UID: "fa0c01c6-ed03-498b-a386-44c85d71e601"). InnerVolumeSpecName "kube-api-access-2zj9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:29:24.036015 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.035974    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqqm7\" (UniqueName: \"kubernetes.io/projected/ca089800-66f3-482f-92db-917bc1868fb0-kube-api-access-tqqm7\") pod \"ca089800-66f3-482f-92db-917bc1868fb0\" (UID: \"ca089800-66f3-482f-92db-917bc1868fb0\") "
Jul 24 18:29:24.036193 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.036116    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca089800-66f3-482f-92db-917bc1868fb0-openshift-service-ca\") pod \"ca089800-66f3-482f-92db-917bc1868fb0\" (UID: \"ca089800-66f3-482f-92db-917bc1868fb0\") "
Jul 24 18:29:24.036193 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.036155    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5b7wv\" (UniqueName: \"kubernetes.io/projected/846247f7-38c6-438b-b04b-8681967a7124-kube-api-access-5b7wv\") pod \"846247f7-38c6-438b-b04b-8681967a7124\" (UID: \"846247f7-38c6-438b-b04b-8681967a7124\") "
Jul 24 18:29:24.036193 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.036174    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca089800-66f3-482f-92db-917bc1868fb0-maas-api-tls\") pod \"ca089800-66f3-482f-92db-917bc1868fb0\" (UID: \"ca089800-66f3-482f-92db-917bc1868fb0\") "
Jul 24 18:29:24.036390 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.036202    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846247f7-38c6-438b-b04b-8681967a7124-plugins-config-volume\") pod \"846247f7-38c6-438b-b04b-8681967a7124\" (UID: \"846247f7-38c6-438b-b04b-8681967a7124\") "
Jul 24 18:29:24.036880 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.036528    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zj9b\" (UniqueName: \"kubernetes.io/projected/fa0c01c6-ed03-498b-a386-44c85d71e601-kube-api-access-2zj9b\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:29:24.036880 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.036535    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca089800-66f3-482f-92db-917bc1868fb0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca089800-66f3-482f-92db-917bc1868fb0" (UID: "ca089800-66f3-482f-92db-917bc1868fb0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:29:24.036880 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.036605    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846247f7-38c6-438b-b04b-8681967a7124-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "846247f7-38c6-438b-b04b-8681967a7124" (UID: "846247f7-38c6-438b-b04b-8681967a7124"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:29:24.038322 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.038294    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca089800-66f3-482f-92db-917bc1868fb0-kube-api-access-tqqm7" (OuterVolumeSpecName: "kube-api-access-tqqm7") pod "ca089800-66f3-482f-92db-917bc1868fb0" (UID: "ca089800-66f3-482f-92db-917bc1868fb0"). InnerVolumeSpecName "kube-api-access-tqqm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:29:24.038402 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.038355    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca089800-66f3-482f-92db-917bc1868fb0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca089800-66f3-482f-92db-917bc1868fb0" (UID: "ca089800-66f3-482f-92db-917bc1868fb0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:29:24.038448 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.038424    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846247f7-38c6-438b-b04b-8681967a7124-kube-api-access-5b7wv" (OuterVolumeSpecName: "kube-api-access-5b7wv") pod "846247f7-38c6-438b-b04b-8681967a7124" (UID: "846247f7-38c6-438b-b04b-8681967a7124"). InnerVolumeSpecName "kube-api-access-5b7wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:29:24.137506 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.137419    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca089800-66f3-482f-92db-917bc1868fb0-openshift-service-ca\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:29:24.137506 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.137449    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b7wv\" (UniqueName: \"kubernetes.io/projected/846247f7-38c6-438b-b04b-8681967a7124-kube-api-access-5b7wv\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:29:24.137506 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.137460    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca089800-66f3-482f-92db-917bc1868fb0-maas-api-tls\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:29:24.137506 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.137468    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846247f7-38c6-438b-b04b-8681967a7124-plugins-config-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:29:24.137506 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.137477    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqqm7\" (UniqueName: \"kubernetes.io/projected/ca089800-66f3-482f-92db-917bc1868fb0-kube-api-access-tqqm7\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:29:24.226801 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.226756    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84"]
Jul 24 18:29:24.231166 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.231131    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2542df-54d5bf758b-tbk84"]
Jul 24 18:29:24.242104 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.242079    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"]
Jul 24 18:29:24.249356 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.249335    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2542df-6764947cb-l6h7l"]
Jul 24 18:29:24.917614 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.917583    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm"
Jul 24 18:29:24.918109 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.917612    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm" event={"ID":"846247f7-38c6-438b-b04b-8681967a7124","Type":"ContainerDied","Data":"63c5f02ee7febe6306cda627a91e3daaa363239b5170ad51f604aadd6309aa87"}
Jul 24 18:29:24.918109 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.917684    2578 scope.go:117] "RemoveContainer" containerID="759ff495741ab57757ced944fc7df40d8f6fe164c95648be72cf241f44861754"
Jul 24 18:29:24.939543 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.939517    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm"]
Jul 24 18:29:24.943535 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:24.943509    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2542df-5b6648855b-9stbm"]
Jul 24 18:29:25.967454 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:25.967422    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="846247f7-38c6-438b-b04b-8681967a7124" path="/var/lib/kubelet/pods/846247f7-38c6-438b-b04b-8681967a7124/volumes"
Jul 24 18:29:25.967831 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:25.967817    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca089800-66f3-482f-92db-917bc1868fb0" path="/var/lib/kubelet/pods/ca089800-66f3-482f-92db-917bc1868fb0/volumes"
Jul 24 18:29:25.968147 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:25.968134    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa0c01c6-ed03-498b-a386-44c85d71e601" path="/var/lib/kubelet/pods/fa0c01c6-ed03-498b-a386-44c85d71e601/volumes"
Jul 24 18:29:28.151628 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:28.151594    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:28.510014 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:28.509925    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:34.650033 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:34.649993    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:35.959000 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:35.958957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:45.663701 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:45.663657    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"]
Jul 24 18:29:45.664150 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:45.663951    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw" podUID="9f32a8b6-14a6-4d7f-b647-0140590a6ce8" containerName="maas-api" containerID="cri-o://3f2a9c7f94c82b02e824b631364db6b73073ff98688749bf34b7ad2cefffb87e" gracePeriod=30
Jul 24 18:29:45.683117 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:45.683092    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:45.718892 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:45.718855    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v"]
Jul 24 18:29:45.719144 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:45.719115    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v" podUID="b116a61e-f964-4bb7-860c-7f3e6a799af9" containerName="payload-pre-processing" containerID="cri-o://249e8049654a33235213b1518b97a392d3494637d7cfe35549cd3e26c6447247" gracePeriod=30
Jul 24 18:29:45.780505 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:45.780477    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:45.881117 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:45.881080    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:45.975116 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:45.975081    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"
Jul 24 18:29:46.005658 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.005633    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v"
Jul 24 18:29:46.033115 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.033083    2578 generic.go:356] "Generic (PLEG): container finished" podID="b116a61e-f964-4bb7-860c-7f3e6a799af9" containerID="249e8049654a33235213b1518b97a392d3494637d7cfe35549cd3e26c6447247" exitCode=0
Jul 24 18:29:46.033115 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.033118    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v"
Jul 24 18:29:46.033362 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.033164    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v" event={"ID":"b116a61e-f964-4bb7-860c-7f3e6a799af9","Type":"ContainerDied","Data":"249e8049654a33235213b1518b97a392d3494637d7cfe35549cd3e26c6447247"}
Jul 24 18:29:46.033362 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.033212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v" event={"ID":"b116a61e-f964-4bb7-860c-7f3e6a799af9","Type":"ContainerDied","Data":"fe5a72c33d852583e219a6b66450563b55147a90dc7b55b2edab5ab5c83aac75"}
Jul 24 18:29:46.033362 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.033240    2578 scope.go:117] "RemoveContainer" containerID="249e8049654a33235213b1518b97a392d3494637d7cfe35549cd3e26c6447247"
Jul 24 18:29:46.034533 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.034512    2578 generic.go:356] "Generic (PLEG): container finished" podID="9f32a8b6-14a6-4d7f-b647-0140590a6ce8" containerID="3f2a9c7f94c82b02e824b631364db6b73073ff98688749bf34b7ad2cefffb87e" exitCode=0
Jul 24 18:29:46.034650 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.034537    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw" event={"ID":"9f32a8b6-14a6-4d7f-b647-0140590a6ce8","Type":"ContainerDied","Data":"3f2a9c7f94c82b02e824b631364db6b73073ff98688749bf34b7ad2cefffb87e"}
Jul 24 18:29:46.034650 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.034562    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"
Jul 24 18:29:46.034650 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.034593    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw" event={"ID":"9f32a8b6-14a6-4d7f-b647-0140590a6ce8","Type":"ContainerDied","Data":"1b472c02955ed5ceae8b284a0c5c4d84ff5ad646ee75f06c31c54a4c8f7c5d3e"}
Jul 24 18:29:46.043248 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.043228    2578 scope.go:117] "RemoveContainer" containerID="249e8049654a33235213b1518b97a392d3494637d7cfe35549cd3e26c6447247"
Jul 24 18:29:46.043523 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:29:46.043499    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249e8049654a33235213b1518b97a392d3494637d7cfe35549cd3e26c6447247\": container with ID starting with 249e8049654a33235213b1518b97a392d3494637d7cfe35549cd3e26c6447247 not found: ID does not exist" containerID="249e8049654a33235213b1518b97a392d3494637d7cfe35549cd3e26c6447247"
Jul 24 18:29:46.043606 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.043531    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249e8049654a33235213b1518b97a392d3494637d7cfe35549cd3e26c6447247"} err="failed to get container status \"249e8049654a33235213b1518b97a392d3494637d7cfe35549cd3e26c6447247\": rpc error: code = NotFound desc = could not find container \"249e8049654a33235213b1518b97a392d3494637d7cfe35549cd3e26c6447247\": container with ID starting with 249e8049654a33235213b1518b97a392d3494637d7cfe35549cd3e26c6447247 not found: ID does not exist"
Jul 24 18:29:46.043606 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.043548    2578 scope.go:117] "RemoveContainer" containerID="3f2a9c7f94c82b02e824b631364db6b73073ff98688749bf34b7ad2cefffb87e"
Jul 24 18:29:46.048802 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.048769    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b116a61e-f964-4bb7-860c-7f3e6a799af9-plugins-config-volume\") pod \"b116a61e-f964-4bb7-860c-7f3e6a799af9\" (UID: \"b116a61e-f964-4bb7-860c-7f3e6a799af9\") "
Jul 24 18:29:46.048902 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.048830    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dls8n\" (UniqueName: \"kubernetes.io/projected/b116a61e-f964-4bb7-860c-7f3e6a799af9-kube-api-access-dls8n\") pod \"b116a61e-f964-4bb7-860c-7f3e6a799af9\" (UID: \"b116a61e-f964-4bb7-860c-7f3e6a799af9\") "
Jul 24 18:29:46.048902 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.048884    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-openshift-service-ca\") pod \"9f32a8b6-14a6-4d7f-b647-0140590a6ce8\" (UID: \"9f32a8b6-14a6-4d7f-b647-0140590a6ce8\") "
Jul 24 18:29:46.049035 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.048916    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r5tj\" (UniqueName: \"kubernetes.io/projected/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-kube-api-access-5r5tj\") pod \"9f32a8b6-14a6-4d7f-b647-0140590a6ce8\" (UID: \"9f32a8b6-14a6-4d7f-b647-0140590a6ce8\") "
Jul 24 18:29:46.049035 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.048943    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-maas-api-tls\") pod \"9f32a8b6-14a6-4d7f-b647-0140590a6ce8\" (UID: \"9f32a8b6-14a6-4d7f-b647-0140590a6ce8\") "
Jul 24 18:29:46.049143 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.049110    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b116a61e-f964-4bb7-860c-7f3e6a799af9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b116a61e-f964-4bb7-860c-7f3e6a799af9" (UID: "b116a61e-f964-4bb7-860c-7f3e6a799af9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:29:46.049289 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.049267    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f32a8b6-14a6-4d7f-b647-0140590a6ce8" (UID: "9f32a8b6-14a6-4d7f-b647-0140590a6ce8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:29:46.049863 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.049590    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b116a61e-f964-4bb7-860c-7f3e6a799af9-plugins-config-volume\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:29:46.049863 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.049617    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-openshift-service-ca\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:29:46.051394 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.051363    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b116a61e-f964-4bb7-860c-7f3e6a799af9-kube-api-access-dls8n" (OuterVolumeSpecName: "kube-api-access-dls8n") pod "b116a61e-f964-4bb7-860c-7f3e6a799af9" (UID: "b116a61e-f964-4bb7-860c-7f3e6a799af9"). InnerVolumeSpecName "kube-api-access-dls8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:29:46.051685 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.051662    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f32a8b6-14a6-4d7f-b647-0140590a6ce8" (UID: "9f32a8b6-14a6-4d7f-b647-0140590a6ce8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:29:46.051947 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.051919    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-kube-api-access-5r5tj" (OuterVolumeSpecName: "kube-api-access-5r5tj") pod "9f32a8b6-14a6-4d7f-b647-0140590a6ce8" (UID: "9f32a8b6-14a6-4d7f-b647-0140590a6ce8"). InnerVolumeSpecName "kube-api-access-5r5tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:29:46.052025 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.051947    2578 scope.go:117] "RemoveContainer" containerID="3f2a9c7f94c82b02e824b631364db6b73073ff98688749bf34b7ad2cefffb87e"
Jul 24 18:29:46.052230 ip-10-0-137-93 kubenswrapper[2578]: E0724 18:29:46.052211    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2a9c7f94c82b02e824b631364db6b73073ff98688749bf34b7ad2cefffb87e\": container with ID starting with 3f2a9c7f94c82b02e824b631364db6b73073ff98688749bf34b7ad2cefffb87e not found: ID does not exist" containerID="3f2a9c7f94c82b02e824b631364db6b73073ff98688749bf34b7ad2cefffb87e"
Jul 24 18:29:46.052307 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.052238    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2a9c7f94c82b02e824b631364db6b73073ff98688749bf34b7ad2cefffb87e"} err="failed to get container status \"3f2a9c7f94c82b02e824b631364db6b73073ff98688749bf34b7ad2cefffb87e\": rpc error: code = NotFound desc = could not find container \"3f2a9c7f94c82b02e824b631364db6b73073ff98688749bf34b7ad2cefffb87e\": container with ID starting with 3f2a9c7f94c82b02e824b631364db6b73073ff98688749bf34b7ad2cefffb87e not found: ID does not exist"
Jul 24 18:29:46.150730 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.150703    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dls8n\" (UniqueName: \"kubernetes.io/projected/b116a61e-f964-4bb7-860c-7f3e6a799af9-kube-api-access-dls8n\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:29:46.150730 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.150726    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r5tj\" (UniqueName: \"kubernetes.io/projected/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-kube-api-access-5r5tj\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:29:46.150730 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.150738    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f32a8b6-14a6-4d7f-b647-0140590a6ce8-maas-api-tls\") on node \"ip-10-0-137-93.ec2.internal\" DevicePath \"\""
Jul 24 18:29:46.359653 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.359620    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"]
Jul 24 18:29:46.363661 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.363633    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw"]
Jul 24 18:29:46.374573 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.374546    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v"]
Jul 24 18:29:46.376250 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:46.376231    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f556c1-7ff9bfc8d-gs28v"]
Jul 24 18:29:47.967913 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:47.967880    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f32a8b6-14a6-4d7f-b647-0140590a6ce8" path="/var/lib/kubelet/pods/9f32a8b6-14a6-4d7f-b647-0140590a6ce8/volumes"
Jul 24 18:29:47.968291 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:47.968258    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b116a61e-f964-4bb7-860c-7f3e6a799af9" path="/var/lib/kubelet/pods/b116a61e-f964-4bb7-860c-7f3e6a799af9/volumes"
Jul 24 18:29:48.955883 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:48.955843    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f556c1-7f8847657b-2s4vw" podUID="9f32a8b6-14a6-4d7f-b647-0140590a6ce8" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.74:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 24 18:29:50.331090 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:50.331052    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:50.718179 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:50.718095    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:56.692931 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:56.692896    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:56.794096 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:56.794064    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:56.957746 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:56.957651    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:57.251308 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:57.251215    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:57.621204 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:29:57.621175    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cc64c6669-ghcws_d8913974-7669-449d-8129-b56ff58193e2/authorino/0.log"
Jul 24 18:30:02.228141 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:02.228091    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:03.180936 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:03.180908    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bf5764cb5-v55w9_2ada65a6-5dbf-472c-90e9-108a5e01948b/manager/0.log"
Jul 24 18:30:03.424370 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:03.424335    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-265d5_35cb65cd-f8ec-4f02-a799-6c0f67c254a7/manager/0.log"
Jul 24 18:30:03.658619 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:03.658591    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbf4bd545-bxczs_5a7ebe25-8c76-432e-ac51-7b8180d3e4bc/copy-manifests/0.log"
Jul 24 18:30:03.665459 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:03.665435    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbf4bd545-bxczs_5a7ebe25-8c76-432e-ac51-7b8180d3e4bc/manager/0.log"
Jul 24 18:30:04.128344 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:04.128312    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-dc545c9bc-5h9hb_58b31e7d-f84a-4bba-8a35-cdb23b3ad3c3/maas-api/0.log"
Jul 24 18:30:04.559144 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:04.559115    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-z5c7x_3805b371-4b1d-427e-bd1e-1568461b89e4/postgres/0.log"
Jul 24 18:30:05.308471 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:05.308436    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cc64c6669-ghcws_d8913974-7669-449d-8129-b56ff58193e2/authorino/0.log"
Jul 24 18:30:05.426338 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:05.426306    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-c78rn_375aec64-0cfe-4df5-a935-abe0f42db406/manager/0.log"
Jul 24 18:30:05.539648 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:05.539610    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-5tlqg_4d0cf1b5-e21d-4eca-bacc-5fc111b0549a/manager/0.log"
Jul 24 18:30:05.758272 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:05.758191    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-4zwpl_a7fbfd17-a32c-4639-b06c-d2b9dfe14d3a/manager/0.log"
Jul 24 18:30:05.996251 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:05.996211    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-mq8wk_f99f537d-8bc5-47e1-9f76-8af38e631178/manager/0.log"
Jul 24 18:30:06.466748 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:06.466717    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xxd6h_7913e838-8015-4554-aa79-6fa18a9d537b/discovery/0.log"
Jul 24 18:30:06.577482 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:06.577444    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56c646dc89-5bnkz_97e8e45e-ee3a-44bf-933e-78c64cf63e36/kube-auth-proxy/0.log"
Jul 24 18:30:07.028485 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:07.028454    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9g7th_992231c7-b15d-4590-a3d5-615ed717052b/payload-processing/0.log"
Jul 24 18:30:07.583756 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:07.583728    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-5j9vz_fee74fad-6891-44b0-b0d4-a8009ec58495/main/0.log"
Jul 24 18:30:07.692490 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:07.692464    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-jh2s2_fbea9b36-a10d-460c-98fe-7c280e7b02ee/main/0.log"
Jul 24 18:30:13.028203 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:13.028164    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:13.101067 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:13.101038    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:15.760769 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:15.760735    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mmrr7_4c849bf6-e357-4dca-8005-56fbbcb8198a/global-pull-secret-syncer/0.log"
Jul 24 18:30:15.920797 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:15.920758    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7b5js_789a3029-0741-46b0-969d-312e2181dbc8/konnectivity-agent/0.log"
Jul 24 18:30:16.074090 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:16.074065    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-93.ec2.internal_b13de3ecc1d0e8f58103000ebb6cab18/haproxy/0.log"
Jul 24 18:30:21.173597 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:21.173568    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cc64c6669-ghcws_d8913974-7669-449d-8129-b56ff58193e2/authorino/0.log"
Jul 24 18:30:21.202349 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:21.202324    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-c78rn_375aec64-0cfe-4df5-a935-abe0f42db406/manager/0.log"
Jul 24 18:30:21.236437 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:21.236411    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-5tlqg_4d0cf1b5-e21d-4eca-bacc-5fc111b0549a/manager/0.log"
Jul 24 18:30:21.465413 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:21.465325    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-4zwpl_a7fbfd17-a32c-4639-b06c-d2b9dfe14d3a/manager/0.log"
Jul 24 18:30:21.606814 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:21.606765    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-mq8wk_f99f537d-8bc5-47e1-9f76-8af38e631178/manager/0.log"
Jul 24 18:30:23.609969 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:23.609941    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-585dbd8bc9-5lsgl_2660087a-bcaa-41c3-b3e3-12070c59fe76/metrics-server/0.log"
Jul 24 18:30:23.665236 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:23.665201    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-npsd7_a2015b61-085f-4832-b797-139af3e5d12b/monitoring-plugin/0.log"
Jul 24 18:30:23.807536 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:23.807502    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pfvzx_556b6001-3830-478a-afd7-9cd9bc3a9926/node-exporter/0.log"
Jul 24 18:30:23.835967 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:23.835929    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pfvzx_556b6001-3830-478a-afd7-9cd9bc3a9926/kube-rbac-proxy/0.log"
Jul 24 18:30:23.861230 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:23.861154    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pfvzx_556b6001-3830-478a-afd7-9cd9bc3a9926/init-textfile/0.log"
Jul 24 18:30:24.166577 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:24.166498    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b6c83c3-f823-49f5-9862-6103e98a970a/prometheus/0.log"
Jul 24 18:30:24.205314 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:24.205291    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b6c83c3-f823-49f5-9862-6103e98a970a/config-reloader/0.log"
Jul 24 18:30:24.242195 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:24.242148    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b6c83c3-f823-49f5-9862-6103e98a970a/thanos-sidecar/0.log"
Jul 24 18:30:24.267367 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:24.267343    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b6c83c3-f823-49f5-9862-6103e98a970a/kube-rbac-proxy-web/0.log"
Jul 24 18:30:24.297511 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:24.297481    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b6c83c3-f823-49f5-9862-6103e98a970a/kube-rbac-proxy/0.log"
Jul 24 18:30:24.327622 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:24.327592    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b6c83c3-f823-49f5-9862-6103e98a970a/kube-rbac-proxy-thanos/0.log"
Jul 24 18:30:24.355675 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:24.355651    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b6c83c3-f823-49f5-9862-6103e98a970a/init-config-reloader/0.log"
Jul 24 18:30:24.431856 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:24.431769    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-k7f9v_532a4ad5-58b1-48b3-b07e-b033f4a7bde4/prometheus-operator/0.log"
Jul 24 18:30:24.478442 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:24.478411    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-k7f9v_532a4ad5-58b1-48b3-b07e-b033f4a7bde4/kube-rbac-proxy/0.log"
Jul 24 18:30:24.591824 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:24.591771    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cd85757d-vlm7b_388ad4ed-8180-4947-b4de-b09e487fc128/telemeter-client/0.log"
Jul 24 18:30:24.620629 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:24.620603    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cd85757d-vlm7b_388ad4ed-8180-4947-b4de-b09e487fc128/reload/0.log"
Jul 24 18:30:24.646002 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:24.645977    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cd85757d-vlm7b_388ad4ed-8180-4947-b4de-b09e487fc128/kube-rbac-proxy/0.log"
Jul 24 18:30:24.680165 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:24.680135    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bc9895498-jf4x5_419c9f71-0a9d-4a42-b11d-e3eebaa895a1/thanos-query/0.log"
Jul 24 18:30:24.706506 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:24.706438    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bc9895498-jf4x5_419c9f71-0a9d-4a42-b11d-e3eebaa895a1/kube-rbac-proxy-web/0.log"
Jul 24 18:30:24.729019 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:24.728995    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bc9895498-jf4x5_419c9f71-0a9d-4a42-b11d-e3eebaa895a1/kube-rbac-proxy/0.log"
Jul 24 18:30:24.756603 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:24.756578    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bc9895498-jf4x5_419c9f71-0a9d-4a42-b11d-e3eebaa895a1/prom-label-proxy/0.log"
Jul 24 18:30:24.785916 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:24.785893    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bc9895498-jf4x5_419c9f71-0a9d-4a42-b11d-e3eebaa895a1/kube-rbac-proxy-rules/0.log"
Jul 24 18:30:24.808481 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:24.808456    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bc9895498-jf4x5_419c9f71-0a9d-4a42-b11d-e3eebaa895a1/kube-rbac-proxy-metrics/0.log"
Jul 24 18:30:26.134996 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.134967    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"]
Jul 24 18:30:26.135456 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135438    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d89a33c-d410-4946-9a3e-6e30eb0786e7" containerName="maas-api"
Jul 24 18:30:26.135456 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135458    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d89a33c-d410-4946-9a3e-6e30eb0786e7" containerName="maas-api"
Jul 24 18:30:26.135589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135468    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7537ded-cf30-4e4a-bb72-11e41af2c87e" containerName="payload-pre-processing"
Jul 24 18:30:26.135589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135474    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7537ded-cf30-4e4a-bb72-11e41af2c87e" containerName="payload-pre-processing"
Jul 24 18:30:26.135589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135492    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0c01c6-ed03-498b-a386-44c85d71e601" containerName="payload-processing"
Jul 24 18:30:26.135589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135499    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0c01c6-ed03-498b-a386-44c85d71e601" containerName="payload-processing"
Jul 24 18:30:26.135589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135505    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="846247f7-38c6-438b-b04b-8681967a7124" containerName="payload-pre-processing"
Jul 24 18:30:26.135589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135510    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="846247f7-38c6-438b-b04b-8681967a7124" containerName="payload-pre-processing"
Jul 24 18:30:26.135589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135516    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f32a8b6-14a6-4d7f-b647-0140590a6ce8" containerName="maas-api"
Jul 24 18:30:26.135589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135521    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f32a8b6-14a6-4d7f-b647-0140590a6ce8" containerName="maas-api"
Jul 24 18:30:26.135589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135529    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b116a61e-f964-4bb7-860c-7f3e6a799af9" containerName="payload-pre-processing"
Jul 24 18:30:26.135589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135534    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b116a61e-f964-4bb7-860c-7f3e6a799af9" containerName="payload-pre-processing"
Jul 24 18:30:26.135589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135539    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21231bc0-059e-4466-b88b-a7eac3530a56" containerName="payload-pre-processing"
Jul 24 18:30:26.135589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135545    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="21231bc0-059e-4466-b88b-a7eac3530a56" containerName="payload-pre-processing"
Jul 24 18:30:26.135589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135553    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb6fb89-9bca-4788-9807-f70d4dc32b2b" containerName="maas-api"
Jul 24 18:30:26.135589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135558    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb6fb89-9bca-4788-9807-f70d4dc32b2b" containerName="maas-api"
Jul 24 18:30:26.135589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135567    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dae8ab73-e781-4365-9fe4-c923d7753955" containerName="maas-api"
Jul 24 18:30:26.135589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135572    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dae8ab73-e781-4365-9fe4-c923d7753955" containerName="maas-api"
Jul 24 18:30:26.135589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135583    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca089800-66f3-482f-92db-917bc1868fb0" containerName="maas-api"
Jul 24 18:30:26.135589 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135588    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca089800-66f3-482f-92db-917bc1868fb0" containerName="maas-api"
Jul 24 18:30:26.136188 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135660    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="846247f7-38c6-438b-b04b-8681967a7124" containerName="payload-pre-processing"
Jul 24 18:30:26.136188 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135669    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f32a8b6-14a6-4d7f-b647-0140590a6ce8" containerName="maas-api"
Jul 24 18:30:26.136188 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135676    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb6fb89-9bca-4788-9807-f70d4dc32b2b" containerName="maas-api"
Jul 24 18:30:26.136188 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135682    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d89a33c-d410-4946-9a3e-6e30eb0786e7" containerName="maas-api"
Jul 24 18:30:26.136188 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135687    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa0c01c6-ed03-498b-a386-44c85d71e601" containerName="payload-processing"
Jul 24 18:30:26.136188 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135692    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b116a61e-f964-4bb7-860c-7f3e6a799af9" containerName="payload-pre-processing"
Jul 24 18:30:26.136188 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135698    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="21231bc0-059e-4466-b88b-a7eac3530a56" containerName="payload-pre-processing"
Jul 24 18:30:26.136188 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135704    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7537ded-cf30-4e4a-bb72-11e41af2c87e" containerName="payload-pre-processing"
Jul 24 18:30:26.136188 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135710    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca089800-66f3-482f-92db-917bc1868fb0" containerName="maas-api"
Jul 24 18:30:26.136188 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.135714    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dae8ab73-e781-4365-9fe4-c923d7753955" containerName="maas-api"
Jul 24 18:30:26.138842 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.138827    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"
Jul 24 18:30:26.141352 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.141331    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jqhx8\"/\"kube-root-ca.crt\""
Jul 24 18:30:26.141492 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.141456    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jqhx8\"/\"openshift-service-ca.crt\""
Jul 24 18:30:26.141552 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.141535    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jqhx8\"/\"default-dockercfg-sqr6w\""
Jul 24 18:30:26.147308 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.147283    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"]
Jul 24 18:30:26.219375 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.219325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b9f486b-87c3-4d4a-bf08-526b3262b996-sys\") pod \"perf-node-gather-daemonset-smhsx\" (UID: \"1b9f486b-87c3-4d4a-bf08-526b3262b996\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"
Jul 24 18:30:26.219375 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.219378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7d79\" (UniqueName: \"kubernetes.io/projected/1b9f486b-87c3-4d4a-bf08-526b3262b996-kube-api-access-c7d79\") pod \"perf-node-gather-daemonset-smhsx\" (UID: \"1b9f486b-87c3-4d4a-bf08-526b3262b996\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"
Jul 24 18:30:26.219610 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.219440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b9f486b-87c3-4d4a-bf08-526b3262b996-proc\") pod \"perf-node-gather-daemonset-smhsx\" (UID: \"1b9f486b-87c3-4d4a-bf08-526b3262b996\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"
Jul 24 18:30:26.219610 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.219475    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b9f486b-87c3-4d4a-bf08-526b3262b996-lib-modules\") pod \"perf-node-gather-daemonset-smhsx\" (UID: \"1b9f486b-87c3-4d4a-bf08-526b3262b996\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"
Jul 24 18:30:26.219610 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.219530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b9f486b-87c3-4d4a-bf08-526b3262b996-podres\") pod \"perf-node-gather-daemonset-smhsx\" (UID: \"1b9f486b-87c3-4d4a-bf08-526b3262b996\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"
Jul 24 18:30:26.320449 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.320395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b9f486b-87c3-4d4a-bf08-526b3262b996-podres\") pod \"perf-node-gather-daemonset-smhsx\" (UID: \"1b9f486b-87c3-4d4a-bf08-526b3262b996\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"
Jul 24 18:30:26.320642 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.320470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b9f486b-87c3-4d4a-bf08-526b3262b996-sys\") pod \"perf-node-gather-daemonset-smhsx\" (UID: \"1b9f486b-87c3-4d4a-bf08-526b3262b996\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"
Jul 24 18:30:26.320642 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.320491    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7d79\" (UniqueName: \"kubernetes.io/projected/1b9f486b-87c3-4d4a-bf08-526b3262b996-kube-api-access-c7d79\") pod \"perf-node-gather-daemonset-smhsx\" (UID: \"1b9f486b-87c3-4d4a-bf08-526b3262b996\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"
Jul 24 18:30:26.320642 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.320525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b9f486b-87c3-4d4a-bf08-526b3262b996-proc\") pod \"perf-node-gather-daemonset-smhsx\" (UID: \"1b9f486b-87c3-4d4a-bf08-526b3262b996\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"
Jul 24 18:30:26.320642 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.320553    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b9f486b-87c3-4d4a-bf08-526b3262b996-lib-modules\") pod \"perf-node-gather-daemonset-smhsx\" (UID: \"1b9f486b-87c3-4d4a-bf08-526b3262b996\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"
Jul 24 18:30:26.320642 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.320598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b9f486b-87c3-4d4a-bf08-526b3262b996-sys\") pod \"perf-node-gather-daemonset-smhsx\" (UID: \"1b9f486b-87c3-4d4a-bf08-526b3262b996\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"
Jul 24 18:30:26.320642 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.320631    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b9f486b-87c3-4d4a-bf08-526b3262b996-proc\") pod \"perf-node-gather-daemonset-smhsx\" (UID: \"1b9f486b-87c3-4d4a-bf08-526b3262b996\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"
Jul 24 18:30:26.320907 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.320598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b9f486b-87c3-4d4a-bf08-526b3262b996-podres\") pod \"perf-node-gather-daemonset-smhsx\" (UID: \"1b9f486b-87c3-4d4a-bf08-526b3262b996\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"
Jul 24 18:30:26.320907 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.320679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b9f486b-87c3-4d4a-bf08-526b3262b996-lib-modules\") pod \"perf-node-gather-daemonset-smhsx\" (UID: \"1b9f486b-87c3-4d4a-bf08-526b3262b996\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"
Jul 24 18:30:26.329913 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.329888    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7d79\" (UniqueName: \"kubernetes.io/projected/1b9f486b-87c3-4d4a-bf08-526b3262b996-kube-api-access-c7d79\") pod \"perf-node-gather-daemonset-smhsx\" (UID: \"1b9f486b-87c3-4d4a-bf08-526b3262b996\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"
Jul 24 18:30:26.449818 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.449712    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"
Jul 24 18:30:26.583251 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:26.583224    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"]
Jul 24 18:30:26.585465 ip-10-0-137-93 kubenswrapper[2578]: W0724 18:30:26.585432    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1b9f486b_87c3_4d4a_bf08_526b3262b996.slice/crio-0869bbf15bbe5ec28ee8f4ece2c75c2c3933342cf2f44b9dfdcebcd5b3991d03 WatchSource:0}: Error finding container 0869bbf15bbe5ec28ee8f4ece2c75c2c3933342cf2f44b9dfdcebcd5b3991d03: Status 404 returned error can't find the container with id 0869bbf15bbe5ec28ee8f4ece2c75c2c3933342cf2f44b9dfdcebcd5b3991d03
Jul 24 18:30:27.167951 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:27.167917    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-kx42n_0a303a45-e8f0-4b53-940b-d22569b73afb/download-server/0.log"
Jul 24 18:30:27.241704 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:27.241663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx" event={"ID":"1b9f486b-87c3-4d4a-bf08-526b3262b996","Type":"ContainerStarted","Data":"afa53530f0d03155dffe1e2a6800bc494508c338eb7a273190cebbcc94199bdc"}
Jul 24 18:30:27.241891 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:27.241725    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx" event={"ID":"1b9f486b-87c3-4d4a-bf08-526b3262b996","Type":"ContainerStarted","Data":"0869bbf15bbe5ec28ee8f4ece2c75c2c3933342cf2f44b9dfdcebcd5b3991d03"}
Jul 24 18:30:27.241891 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:27.241824    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"
Jul 24 18:30:27.259432 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:27.259382    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx" podStartSLOduration=1.259367141 podStartE2EDuration="1.259367141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:30:27.25753928 +0000 UTC m=+2849.780897757" watchObservedRunningTime="2026-07-24 18:30:27.259367141 +0000 UTC m=+2849.782725619"
Jul 24 18:30:28.830330 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:28.830296    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9jtqg_8929ae32-e84a-4a3b-b7ad-5ee3f7575298/dns/0.log"
Jul 24 18:30:28.851422 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:28.851395    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9jtqg_8929ae32-e84a-4a3b-b7ad-5ee3f7575298/kube-rbac-proxy/0.log"
Jul 24 18:30:28.999795 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:28.999750    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mn4d8_afc9b6f4-a944-4228-97db-a0c9bbc70a7f/dns-node-resolver/0.log"
Jul 24 18:30:29.561427 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:29.561394    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-657dd9f654-2fvxt_3692f369-8fc2-4b2f-b064-034d5f50957c/registry/0.log"
Jul 24 18:30:29.583950 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:29.583918    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bnjwn_65c7cb53-638b-4934-9a59-ca69c44a4373/node-ca/0.log"
Jul 24 18:30:30.759757 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:30.759725    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xxd6h_7913e838-8015-4554-aa79-6fa18a9d537b/discovery/0.log"
Jul 24 18:30:30.783646 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:30.783617    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56c646dc89-5bnkz_97e8e45e-ee3a-44bf-933e-78c64cf63e36/kube-auth-proxy/0.log"
Jul 24 18:30:30.924141 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:30.924109    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9g7th_992231c7-b15d-4590-a3d5-615ed717052b/payload-processing/0.log"
Jul 24 18:30:31.849743 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:31.849711    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x95l7_9f1aee3c-1324-4eaa-8201-95c26602d6c1/serve-healthcheck-canary/0.log"
Jul 24 18:30:32.445561 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:32.445526    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pfqcd_d8d0a6d2-c541-427e-b1e2-48fa4db65735/kube-rbac-proxy/0.log"
Jul 24 18:30:32.468628 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:32.468598    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pfqcd_d8d0a6d2-c541-427e-b1e2-48fa4db65735/exporter/0.log"
Jul 24 18:30:32.498437 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:32.498413    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pfqcd_d8d0a6d2-c541-427e-b1e2-48fa4db65735/extractor/0.log"
Jul 24 18:30:33.256542 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:33.256513    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-smhsx"
Jul 24 18:30:35.357979 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:35.357887    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bf5764cb5-v55w9_2ada65a6-5dbf-472c-90e9-108a5e01948b/manager/0.log"
Jul 24 18:30:35.468173 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:35.468138    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-265d5_35cb65cd-f8ec-4f02-a799-6c0f67c254a7/manager/0.log"
Jul 24 18:30:35.561438 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:35.561404    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbf4bd545-bxczs_5a7ebe25-8c76-432e-ac51-7b8180d3e4bc/manager/0.log"
Jul 24 18:30:35.586477 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:35.586446    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbf4bd545-bxczs_5a7ebe25-8c76-432e-ac51-7b8180d3e4bc/copy-manifests/0.log"
Jul 24 18:30:36.750057 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:36.749980    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xxd6h_7913e838-8015-4554-aa79-6fa18a9d537b/discovery/0.log"
Jul 24 18:30:36.767026 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:36.766998    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56c646dc89-5bnkz_97e8e45e-ee3a-44bf-933e-78c64cf63e36/kube-auth-proxy/0.log"
Jul 24 18:30:36.854447 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:36.854411    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9g7th_992231c7-b15d-4590-a3d5-615ed717052b/payload-processing/0.log"
Jul 24 18:30:38.513456 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:38.513418    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xxd6h_7913e838-8015-4554-aa79-6fa18a9d537b/discovery/0.log"
Jul 24 18:30:38.525595 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:38.525566    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56c646dc89-5bnkz_97e8e45e-ee3a-44bf-933e-78c64cf63e36/kube-auth-proxy/0.log"
Jul 24 18:30:38.630740 ip-10-0-137-93 kubenswrapper[2578]: I0724 18:30:38.630709    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9g7th_992231c7-b15d-4590-a3d5-615ed717052b/payload-processing/0.log"